Decision models are getting more attention for making fast, inexpensive, AI-driven judgments. You give them a question and some information, and they return a score you can use to make a decision. That makes them interesting for tasks like classifying documents or checking an agent’s work.

I’ve been curious about what decision models could do for science and research. So I built a small benchmark around a problem we have at @dair_ai.

Our library holds thousands of AI papers, and we group the best ones into reading collections like Harness Engineering and Self-Improving Agents. Deciding which papers belong takes careful reading of each abstract and other metadata.

So we started with 250 papers we labeled before seeing model predictions. The model sees only the title and abstract and answers two yes-or-no questions, one per collection, each with a confidence score.

Question 1. Is this paper mainly about the harness around a model, such as its prompts, context, memory, tools, agent loops, or multi-agent setup?

Question 2. Is this paper mainly about an agent that improves itself through a loop that makes lasting changes to its weights, prompts, memory, skills, or code?

I started benchmarking this with Drex Max (I got early access to it) from Nace and tested it against the new Jev 1.14.

Both models received the same instructions. We used 101 papers to choose a separate decision threshold for each model and collection, then measured results on the other 149. We selected thresholds by F1, which balances precision and recall, and kept them fixed when scoring the test papers. For your own application, choose thresholds based on which mistakes are more costly.

On the 149 test papers, Drex Max got both labels right 76.5% of the time, compared with 74.5% for Jev 1.14. Across five repeated runs, the averages were 78.1% and 74.9%.

Drex Max also ranked Harness Engineering papers slightly better, with an AUROC of 0.924 against 0.907 for Jev.

Our initial checks suggest that Drex Max benefits from choosing decision thresholds using labeled examples, making threshold tuning a useful step when applying it to your own task.

For this task, it was important to clearly define what counts as a yes before tuning thresholds. If you use decision models for similar tasks, include borderline examples in your labeled set, then test the chosen threshold on separate papers.