Nobody at Rox writes evals anymore. Devin now QAs our agent every week, on real traffic. It reads real conversations with our agent, rebuilds each failure in a full copy of Rox, and turns it into an eval. We just review the PR. The eval suite already covers 72% of real usage happening in the product, vs. 43–57% for the evals we used to write by hand. And it’s growing week over week with Devin. Built with @cognition and @braintrust: