about
Evaluating Genuine Reasoning in LLMs via Esoteric Programming Languages (arxiv.org)
1 point by kerneis 198 days ago | hide | past | pdf | 1 comment on HN

In plain words: A benchmark rewrites 80 standard coding problems into five obscure, rarely-seen programming languages, testing whether models can solve familiar logic in languages they barely encountered during training. Top models solved 100% of the problems in Python or JavaScript but just 0–11% in the obscure ones, and extra examples or self-checking did not close the gap.

Abstract · EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages

Large language models achieve near-ceiling performance on code generation benchmarks, yet most of the programming languages used by popular benchmarks such as SWE-bench and HumanEval (e.g. Python, JavaScript) are squarely in-distribution. They appear at scale in pre-training corpora and are heavily reinforced during post-training. To study LLM performance on unfamiliar programming languages, we introduce EsoLang-Bench, a benchmark using five esoteric programming languages (Brainfuck, Befunge-98, Whitespace, Unlambda, and Shakespeare). All five of our chosen esoteric languages are Turing-complete, so the same algorithmic problems that are solvable in Python or JavaScript are in principle solvable in each of them. Yet, they are unfamiliar to LLMs which makes them a good proxy for evaluating out-of-distribution performance. The unfamiliarity of esoteric languages comprises of: (i) the hard-by-design primitives comprising the language; (ii) substantially less representation in pre-training corpora (340x to over 60,000x fewer public GitHub repositories than Python); (iii) negligible deployment value, which makes targeted inclusion in post-training data economically irrational. We evaluate five frontier models across five prompting strategies and find a dramatic capability gap. The same 80 problems expressed in Python or JavaScript reach 100% accuracy on top frontier models, while the equivalent esoteric versions score only 0-11%. Few-shot learning and self-reflection also fail to close this gap. EsoLang-Bench therefore provides a contamination-resistant testbed for measuring how well frontier models generalise algorithmic problem-solving to programming languages outside their training distribution.

Aman Sharma, Paras Chopra
arXiv:2603.09678 · cs.AI, cs.LG, cs.SE · submitted Mar 10, 2026 · updated May 11, 2026
abstract · pdf · html · 45 pages, 8 figures, preprint

add comment on HN