about
Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers (arxiv.org)
3 points by simonpure 129 days ago | hide | past | pdf | discuss on HN

In plain words: Six AI models tackled 24 tasks under three levels of rules-and-tools wrapper, in 432 trials testing whether stronger models need less structure. Stricter rules cut one top chat model's success 29-38 points, while a top reasoning model did best (91.7%) with the strictest.

Abstract · It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers

A prevalent assumption in LLM agent deployment holds that more structured harnesses universally improve reliability, and that higher-capability models need proportionally less structural guidance -- together implying a monotone inverse relationship between model capability tier and optimal harness complexity. We test this hypothesis through a controlled 432-run experiment crossing six models across four capability tiers with three harness conditions (light, balanced, strict) on HEAT-24, a 24-task synthetic benchmark with git-based workspace verification. Our results refute the monotone inverse relationship on two fronts. First, for the frontier chat model evaluated (Gemini 2.5 Flash), increased harness verbosity lowers VTSR by 29-38 percentage points -- a harness-complexity paradox. Second, for the frontier reasoning model evaluated (Qwen3.5-122B, extended thinking enabled), strict harness achieves the highest VTSR (91.7%) and the lowest latency, the opposite of the prediction. Within the constrained tier, a 2B model (Gemma4:e2B) matches strong-open-tier stability at 91.7% across all harnesses. Because each tier is represented by a single model in this study, these results should be interpreted as model-specific observations; harness sensitivity appears non-monotone across the models evaluated, and depends critically on model type (chat vs. reasoning). We introduce a six-label failure taxonomy showing that format_violation dominates capable-model failures while wrong_file dominates low-capability failures, and we derive practical tier-aware harness selection guidelines.

Yong-eun Cho
arXiv:2605.26731 · cs.AI, cs.CL · submitted May 26, 2026
abstract · pdf · html · 9 pages, 3 figures

add comment on HN