In plain words: AI agents should be trained in automation games like Factorio, where players design and grow factories over time, to build the skills big engineering projects need. Unlike static test tasks, these games force agents to handle uncertain trade-offs and adapt as things change.
Abstract
Continuing advances in frontier model research are paving the way for widespread deployment of AI agents. Meanwhile, global interest in building large, complex systems in software, manufacturing, energy and logistics has never been greater. Although AI driven system engineering holds tremendous promise, the static benchmarks dominating agent evaluations today fail to capture the crucial skills required for implementing dynamic systems, such as managing uncertain trade-offs and ensuring proactive adaptability. This position paper advocates for training and evaluating AI agents' system engineering abilities through automation-oriented sandbox games-particularly Factorio. By directing research efforts in this direction, we can equip AI agents with the specialized reasoning and long-horizon planning necessary to design, maintain, and optimize tomorrow's most demanding engineering projects.
Neel Kant
arXiv:2502.01492 · cs.AI · submitted Feb 3, 2025
abstract · pdf · html