Press Space to continue
AI benchmarks underestimate model capabilities by 82 percent
Press Space to continue
Benchmarks have been lying to us about how good AI actually is. By 82%. A new paper just quantified something everyone in AI has felt but never measured: the model you are using is nowhere near the ceiling of what's actually achievable today. Here's the setup. Every benchmark you have ever seen reports one model, one run. But that's not how capability actually works: → Different models are good at different things → The same model gives different answers on different runs → Somewhere in…
withmartian@withmartianSep 3
We got 46% fewer errors than the single best LLM across the 16 most used benchmarks (TerminalBench, LiveCodeBench, etc). Here's how that's possible and what each model can achieve when used optimally (every benchmarks misses the majority of model capabilities) 👇 Interactive Site: https://t.co/6ZqljIhKBh Academic Paper: https://t.co/AX7QEMtEF5
