Press Space to continue
Production judges find failures offline datasets miss
Press Space to continue
the useful part of a judge on production traffic is what happens to the outputs it flags calibrate it on human labels, flag low scorers, confirm the real failures, add them to evals, fix, rerun prod finds the failures your offline dataset never had
PyTorch@PyTorch4d ago
Now: @Shopify’s continual learning loop with PyTorch and @vLLM_project takes the stage at #PyTorchCon 2026 with Cody Mazza-Anthony @cmazzaanthony. Using Buyer Profile as an example, Cody walks through how a calibrated judge flags low-scoring outputs in production. Critics diagnose each failure, and a replay with a critic hint is scored again by the judge. Accepted repairs expand and improve the training data. Cody shows how a small specialist model can outperform a frontier model on this task, then turns to optimizing inference through prompt compression. Watch the keynote livestream: https://t.co/6TtaEkw0x7