Its getting hard to benchmark frontier agent performance on | Tech Twitter

Finding signal on Twitter is more difficult than it used to be. We curate the best tweets on topics like AI, startups, and product development every weekday so you can focus on what matters.

Trendingwhat is spiking in the last 24 hours Time Travelreplay any day in the archive Topicsclusters the feed keeps returning to Profilesthe accounts we watch

//

Articleslong-form writeups pulled from the feed Threadspublished editions, start to finish Judged by Jevone post at a time, with a verdict

//

Agentshow machines read this site Skillsinstallable skills for your agent agents.mdthe plain-text brief

//News//About//

Focus

Press Space to continue

Built by Daniel Hunter

Its getting hard to benchmark frontier agent performance on | Tech Twitter

Press Space to continue

Its getting hard to benchmark frontier agent performance on longer tasks. Repeated measurement is very expensive and there are differences between using models in harnesses versus via APIs. I suspect benchmarks understate progress, they are built for models, not harnessed agents

Judged by Jev

Post analysis

Analyzing this post

A rubric-based AI judgment,
not ground truth.

Previous tweet Next tweet More from @emollick