Press Space to continue
Press Space to continue
Google DeepMind researcher @iamtrask says the independent-evaluator vibe shift is only step one because frontier AI ultimately needs thousands of outsiders probing it continuously: "It's a meaningful step forward, but it's not getting us all the way to the finish." "Sometimes just using the public products themselves is not enough to answer all the questions that we need. You need access to APIs that might not have been built for public consumption." "It's fundamentally a low-scale approach.…
MTS@MTSlive7h ago
FULL INTERVIEW: Andrew Trask says AI won't end with one giant model. He expects millions of models, ensembled and routed per prompt, to beat any single frontier model on quality and price, which would make AI look more like the PC and the internet than the mainframe. @iamtrask is a senior research scientist at @GoogleDeepMind and the founder of @openminedorg, which helped run the first double-blind evaluation of a frontier model. He joined @theojaffee to lay out who actually gets to audit the labs: 00:40 why embedded evaluators aren't enough 03:54 the "evil EAs" criticism of eval orgs 05:21 why labs only call people they already trust 08:22 a day in the life of an independent evaluator 10:10 how many evaluators belong inside the labs 12:09 will models just game the evals 13:42 the first double-blind eval on a frontier model 15:53 OpenAI's math release and sandboxes with no holes 20:40 why defining what's good could become a job 22:14 the case against one giant model winning 24:32 the proof from OpenRouter and Sakana
View