about
Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents (arxiv.org)
3 points by gfto on May 28, 2025 | hide | past | pdf | discuss on HN

In plain words: A simulated vending machine business—stocking, ordering, pricing, paying fees—tests whether an AI agent stays coherent across months of decisions. Two models usually profited, but every model had runs that derailed, and failures did not line up with its memory filling up.

Abstract

While Large Language Models (LLMs) can exhibit impressive proficiency in isolated, short-term tasks, they often fail to maintain coherent performance over longer time horizons. In this paper, we present Vending-Bench, a simulated environment designed to specifically test an LLM-based agent's ability to manage a straightforward, long-running business scenario: operating a vending machine. Agents must balance inventories, place orders, set prices, and handle daily fees - tasks that are each simple but collectively, over long horizons (>20M tokens per run) stress an LLM's capacity for sustained, coherent decision-making. Our experiments reveal high variance in performance across multiple LLMs: Claude 3.5 Sonnet and o3-mini manage the machine well in most runs and turn a profit, but all models have runs that derail, either through misinterpreting delivery schedules, forgetting orders, or descending into tangential "meltdown" loops from which they rarely recover. We find no clear correlation between failures and the point at which the model's context window becomes full, suggesting that these breakdowns do not stem from memory limits. Apart from highlighting the high variance in performance over long time horizons, Vending-Bench also tests models' ability to acquire capital, a necessity in many hypothetical dangerous AI scenarios. We hope the benchmark can help in preparing for the advent of stronger AI systems.

Axel Backlund, Lukas Petersson
arXiv:2502.15840 · cs.AI · submitted Feb 20, 2025
abstract · pdf · html

add comment on HN
Also discussed: Jun 2025 (1 point, 0 comments) · Jun 2025 (3 points, 0 comments) · May 2025 (2 points, 0 comments) · May 2025 (4 points, 1 comment) · May 2025 (2 points, 0 comments) · Apr 2025 (14 points, 0 comments)