Press Space to continue
Benchmarks Need Ephemeral Specificity to Matter
Press Space to continue
who benchmarks the benchmarkers? a good eval is: - ephemeral. it snapshots a moment in time and capabilities. - grounded. it directly measures performance on tasks and data you care about. - specific. it operates on the desired results and not some proxy of them. take notes!
@Oct 15
Typesafe's CEO @CompleteSkeptic does not trust public benchmarks. Neither do I, Diogo. So @p0 and I wrote a blog evaluating 8 decision models on real company training datasets we have personally audited and iterated on. For what it's worth though, Jev is still on top 👑 https://t.co/Qd2g0Hwp4V