about
How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering? (arxiv.org)
2 points by tchalla 23 days ago | hide | past | pdf | discuss on HN

In plain words: They compared elaborate multi-agent setups that orchestrate retrieval and planning against one simple coding agent that just reads, writes, and runs commands, using the same model and time limit. The simple agent matched the fancy setups, showing the model itself drives performance.

Abstract

Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards. Often motivated by progress stagnation over long-horizon cycles and limited Large Language Model (LLM) primitives, modern MLE agents are deployed on top of increasingly elaborate machinery: multi-agent orchestrators, dedicated retrieval subagents, and more. While such harnesses expand, the use of more primitive but improved coding agents - where LLMs have direct access to the execution environment through read, write, and bash primitives - has received little attention in the field. In this paper we find that, under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance. Via a series of large-scale systematic ablation studies, we argue that the machinery layers become redundant in the coding agent setting. We conclude that the effort spent elaborating hand-crafted harnesses around strong models yields poor returns for current MLE benchmarks.

Kirill Brilliantov, Alejandro Hernández-Cano, Emmanuel Abbé
arXiv:2609.40303 · cs.AI · submitted Sep 30, 2026
abstract · pdf · html

add comment on HN
Also discussed: Oct 2026 (2 points, 0 comments) · Oct 2026 (1 point, 0 comments) · Oct 2026 (3 points, 0 comments)