This is brilliant. Recommended reading. It's a great example of

Press Space to continue

This is brilliant. Recommended reading. It's a great example of how you could potentially build more effective harnesses using System One and System Two models. It explores Jev's potential as a fuzzy linter. Before you dismiss it, check out the setup. I like this post because it provides a few details of a good setup for testing these ideas. It scopes the problem well, like filtering out rules that are easy to judge (i.e., don't require extra context/resoning). That's important because…

MichaelThiessen@MichaelThiessen1d ago

I’ve been experimenting a bit with Jev as a fuzzy linter that runs after edits in your agent harness: looks very promising so far in my evals. First I converted all of our coding guidelines to tiny rules that are very easy to judge. Then, only kept the ones that don’t require extra context/files/reasoning. I want it to be a single call, super fast, and Jev isn’t smart enough to handle complex or ambiguous rules. I then created a synthetic eval that has different kinds of tool calls to exercise and test every one of those rules. Also a held out set that we don’t train on so we don’t accidentally overfit (ie benchmax). One cool thing that seems to work well: Jev returns confidence, so in addition to high confidence violations, we can also return medium confidence and get the agent to double check. Adding that lower confidence tier seems to double what it catches, and only increases false positives a little. I still have yet to test this inside an actual agent to see how the agent reacts to this feedback. Hopefully next week I can get a decent eval going and see how well this works E2E.