Press Space to continue
Decision models work well as LLM judges
Press Space to continue
Learn to build evals, folks. Highly recommended reading on using decision models for evals. From our internal evals, System One models like Jev are a great fit for LLM Judges. If you want to go beyond Hamel's great write-up, I've also put together a few resources on applying decision models to LLM judges. (bookmark it) 1) Jev-as-a-Judge for Agent Evaluations - https://academy.dair.ai/dashboard/resour… 2) Jev-as-a-Judge: Evaluating Agents with a Decision Model -…
HamelHusain@HamelHusain4d ago
New blog post 🎉: How To Use Decision Models For Evals In this post, we show you how to choose a decision model for your task, tune its threshold (don't assume the 0.5 default is good!), and interpret its scores. Lots of cool interactive charts here. https://t.co/uYAfte9Kb5