about
The Path Not Taken: Duality in Reasoning about Program Execution (arxiv.org)
2 points by PaulHoule 145 days ago | hide | past | pdf | discuss on HN

In plain words: A benchmark tests code understanding two ways: predict what a program does with a given input, and change the input to make it do something specific. Testing 13 language models on 445 pairs showed this two-way check reliably measures real understanding of code execution.

Abstract

Large language models (LLMs) have shown remarkable capabilities across diverse coding tasks. However, their adoption requires a true understanding of program execution rather than relying on surface-level patterns. Existing benchmarks primarily focus on predicting program properties tied to specific inputs (e.g., code coverage, program outputs). As a result, they provide a narrow view of dynamic code reasoning and are prone to data contamination. We argue that understanding program execution requires evaluating its inherent duality through two complementary reasoning tasks: (i) predicting a program's observed behavior for a given input, and (ii) inferring how the input must be mutated toward a specific behavioral objective. Both tasks jointly probe a model's causal understanding of execution flow. We instantiate this duality in DexBench, a benchmark comprising 445 paired instances, and evaluate 13 LLMs. Our results demonstrate that dual-path reasoning provides a robust and discriminative proxy for dynamic code understanding.

Eshgin Hasanov, Md Mahadi Hassan Sibat, Santu Karmaker, Aashish Yadavally
arXiv:2604.20917 · cs.LG, cs.AI, cs.CL, cs.PL, cs.SE · submitted Apr 22, 2026
abstract · pdf · html · Accepted to ACL 2026 Main Conference

add comment on HN