about
Benchmarking Coding Agents Under Evolving Requirements (arxiv.org)
5 points by DeathArrow 1 day ago | hide | past | pdf | discuss on HN

In plain words: A test set gives assistants 61 tasks where the user keeps adding requirements and reporting failures, with checks that track new behavior, repairs, and whether old features work. The best finished 72.1% of tasks, but only 15.4% of failed steps recovered within two fixes.

Abstract · Benchmarking Coding Agents under Evolving Requirements

Software development with coding agents often begins with a rough request that users refine after trying the implementation. Final correctness alone does not capture the repeated guidance this process requires. We introduce FeatureLoop, a benchmark of 61 human-reviewed tasks from 34 repositories across five primary programming languages and seven software forms. Each task follows one development goal in a persistent workspace through evolution feedback, which adds or clarifies requirements, and completion feedback, which reports failures of stated requirements. Cumulative executable checks track new behavior, repair, and preservation of earlier functionality. We evaluate 12 frontier models and find that the strongest completes 72.1% of tasks with both feedback types. Across models, however, only 15.4% of observed failed stages recover within two corrective revisions. Further analysis reveals incomplete repairs, regressions, and model-specific differences in feedback use and token efficiency, identifying obstacles to reliable, low-intervention development.

Jia Feng, Alyssa Wu, Cuiyun Gao, Yunlong Wang, Songyuanyi Lu, Zhichao Hu
arXiv:2610.35488 · cs.SE, cs.LG · submitted Oct 26, 2026 · updated Oct 29, 2026
abstract · pdf · html · 51 pages, including appendices

add comment on HN