yet another deeply unsaturated benchmark on model’s ability to

Press Space to continue

yet another deeply unsaturated benchmark on model’s ability to maintain a codebase

Can LMs discover & fix bugs if you don't tell them what went wrong? To proactively maintain a repo, agents need to find problems before users do. SWE-sweep benchmarks this on 100 repos, 22 languages, 4k real bugs (numpy, php interpreter, lean kernel). Top models get <5%. https://t.co/OwqrjHk64q