elvis@omarsar02d ago

📝 Decision Models for Research Triage Decision models are getting more attention for making fast, inexpensive, AI-driven judgments. You give them a question and some information, and they return a score you can use to make a decision. That makes them interesting for tasks like classifying documents or checking an agent’s work. I’ve been curious about what decision models could do for science and research. So I built a small benchmark around a problem we have at @dair_ai. Our library holds thousands of AI papers, and we group the best ones into reading collections like Harness Engineering and Self-Improving Agents. Deciding which papers belong takes careful reading of each abstract and other metadata. So we started with 250 papers we labeled before seeing model predictions. The model sees only the title and abstract and answers two yes-or-no questions, one per collection, each with a confidence score. Question 1. Is this paper mainly about the harness around a model, such as its prompts, context, memory, tools, agent loops, or multi-agent setup? Question 2. Is this paper mainly about an agent that improves itself through a loop that makes lasting changes to its weights, prompts, memory, skills, or code? I started benchmarking this with Drex Max (I got early access to it) from Nace and tested it against the new Jev 1.14. Both models received the same instructions. We used 101 papers to choose a separate decision threshold for each model and collection, then measured results on the other 149. We selected thresholds by F1, which balances precision and recall, and kept them fixed when scoring the test papers. For your own application, choose thresholds based on which mistakes are more costly. On the 149 test papers, Drex Max got both labels right 76.5% of the time, compared with 74.5% for Jev 1.14. Across five repeated runs, the averages were 78.1% and 74.9%. Drex Max also ranked Harness Engineering papers slightly better, with an AUROC of 0.924 against 0.907 for Jev. Our initial checks suggest that Drex Max benefits from choosing decision thresholds using labeled examples, making threshold tuning a useful step when applying it to your own task. For this task, it was important to clearly define what counts as a yes before tuning thresholds. If you use decision models for similar tasks, include borderline examples in your labeled set, then test the chosen threshold on separate papers. My takeaway is that decision models look useful for research triage when questions are well defined, and thresholds are tested on labeled examples. Threshold tuning matters, but so does the quality of the labeled set. This is still an exploratory comparison with a small number of labels and papers used during earlier development. Freshly labeled papers would give us a stronger test before we rely on these decisions. But after this initially exploratory work, we are confident we can continue scaling and improving our benchmarks to apply these decision models in production. Get started with Drex Max here: https://console.nace.ai/docs/drex/guides/drex-max Console: https://console.nace.ai/login Thanks to @NaceAI for the credits and partnership on this post. https://x.com/i/article/2113280256119312384