about
AetherCode: Evaluating LLMs' Ability to Win in Premier Programming Competitions (arxiv.org)
1 point by limoce on Aug 25, 2025 | hide | past | pdf | discuss on HN

In plain words: A new set of programming contest problems, taken from top events like the IOI and ICPC, tests AI coding with harder questions and expert-checked test cases. On these, models fall well short of elite human competitors, unlike what easier, poorly tested benchmarks suggest.

Abstract · AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitions

Competitive programming has emerged as a critical benchmark for evaluating the reasoning and coding capabilities of Large Language Models (LLMs). Despite impressive progress on existing benchmarks, we argue that current evaluations overstate model proficiency, masking a substantial gap between LLMs and elite human programmers. This gap arises from two key limitations: insufficient difficulty and scope of benchmark problems, and evaluation bias from low-quality test cases. To address these shortcomings, we present AetherCode, a new benchmark that draws problems from premier programming competitions such as IOI and ICPC, offering broader coverage and higher difficulty. AetherCode further incorporates comprehensive, expert-validated test suites built through a hybrid of automated generation and human curation, ensuring rigorous and reliable assessment. By combining challenging problem design with robust evaluation, AetherCode provides a more faithful measure of LLM capabilities and sets a new standard for future research in code reasoning.

Zihan Wang, Jiaze Chen, Zhicheng Liu, Markus Mak, Yidi Du, Geonsik Moon, Luoqi Xu, Aaron Tua, Kunshuo Peng, Jiayi Lu, Mingfei Xia, Boqian Zou, et al.
arXiv:2508.16402 · cs.SE, cs.CL · submitted Aug 22, 2025
abstract · pdf · html · 15 pages

add comment on HN
Also discussed: Aug 2025 (2 points, 1 comment)