arkyyang@arkyyang6h ago

Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnesses — new paper (arXiv, 4 Sep 2026; accepted EMNLP 2026). https://t.co/pKJEV3lx5R Treats harness optimization around a fixed model as budgeted selection: edits are guarded intercepts at the tool boundary, not rewrites of execution logic. PRISM clusters failures, routes each repair to a prompt, middleware, or joint surface, and picks candidates by gate pass-rate AND reliability (RelLift95), not score alone. 3 distinct takeaways for product builders: - Report selected-harness reliability, not just mean lift. BH found the best individual harnesses but had negative reliable lift on both tau-bench domains — a strong explorer, an unreliable selector. - Constrain middleware to tool-boundary edit patterns (silent correction, error blocking, prerequisite blocking). Unconstrained middleware (BH-MW) fell below its own prompt-only variant on Retail; the constraint, not the surface, is what makes middleware safely searchable. - Route each failure to the right surface: prompts for global policy, middleware for locally-checkable execution invariants. Removing routing collapsed lift first-order on all three tasks; prompts alone miss runtime failures, unconstrained middleware adds selection risk. Experiment setup (from the paper): fixed inner execution model gpt-5.4-mini, outer proposer claude-opus-4.7. Benchmarks: BFCL multi-round (600-case scorecard) and tau2-Retail / tau2-Telecom (74-case scorecard each of 114 tasks). Baselines BH, GEPA, MIPROv2 + PRISM; disjoint repair/gate/scorecard splits, 4 seeds, up to 16 runs. Baseline = native Deep Agents (v0.5.6). Metrics: mean held-out lift (pp), WorstLift, repeatability, RelLift95(B) by 5,000 bootstrap. PRISM-MW mean lifts 14.2 / 14.9 / 10.1 pp and positive RelLift95 on all three.