about
EnvTrace: Simulation-Based Semantic Evaluation of LLM Code (arxiv.org)
1 point by amscotti 324 days ago | hide | past | pdf | discuss on HN

In plain words: A simulated copy of a beamline runs AI-written control code and compares its step-by-step actions with the right ones, instead of just checking final outputs like unit tests do. Across more than 30 AI models, many of the best came close to human-level code.

Abstract · EnvTrace: Simulation-Based Semantic Evaluation of LLM Code via Execution Trace Alignment -- Demonstrated at Synchrotron Beamlines

Evaluating large language models (LLMs) for instrument control requires methods that go beyond standard, stateless algorithmic benchmarks, since the behavior of physical systems cannot be fully captured by unit tests alone. Here we introduce EnvTrace, a simulation-based method that evaluates execution traces to assess semantic code equivalence. EnvTrace is demonstrated with a beamline control-logic digital twin to facilitate the evaluation of instrument control code, with the digital twin itself also enabling the pre-execution validation of live experiments. Over 30 LLMs were evaluated using trace alignment to generate a multi-faceted score for functional correctness across key behavioral dimensions, showing that many top-tier models can approach human-level performance in rapid control-code generation. This is a first step toward a broader vision where LLMs and digital twins work symbiotically: LLMs providing intuitive control and agentic orchestration, and digital twins offering safe and high-fidelity environments, paving the way towards autonomous embodied AI.

Noah van der Vleuten, Anthony Flores, Shray Mathur, Max Rakitin, Thomas Hopkins, Kevin G. Yager, Esther H. R. Tsai
arXiv:2511.09964 · cs.SE, cs.AI, cs.PL · submitted Nov 13, 2025
abstract · pdf · html

add comment on HN