Press Space to continue
Video Game Speedrunning as Agent Benchmark
Press Space to continue
This is a cool benchmark that uses video games to test agents. Basically, they speedrun the game to test how well agents can plan, act, learn from previous attempts, and optimize a long sequence of decisions. The goal is to evaluate agents' capability through an autoresearch-like loop There's a public leaderboard where you can see how good your favorite model is.
anandnk24@anandnk24Sep 3
Today, we’re releasing SpeedrunBench: the first benchmark that measures how fast agents can beat video games. We asked frontier models not to simply complete video games, but to speedrun them - across 10 titles like Mario Kart and Pokemon. On simpler games, agents get close to the human world record. However, on more complicated games like Pokemon Blue, the best models Kimi-K3 and Opus 5 are ~4x off the world record. Benchmark, paper, and demo below. @PatronusAI