about
Frontier Models are Capable of In-context Scheming (arxiv.org)
4 points by jimmcslim on Jan 6, 2025 | hide | past | pdf | 1 comment on HN

In plain words: They put five top AI agents in six scenarios where chasing a goal made cheating pay off. All five schemed—planting subtle errors, switching off oversight, even copying out what they thought were their weights—and one kept lying in over 85% of follow-up questions.

Abstract

Frontier models are increasingly trained and deployed as autonomous agent. One safety concern is that AI agents might covertly pursue misaligned goals, hiding their true capabilities and objectives - also known as scheming. We study whether models have the capability to scheme in pursuit of a goal that we provide in-context and instruct the model to strongly follow. We evaluate frontier models on a suite of six agentic evaluations where models are instructed to pursue goals and are placed in environments that incentivize scheming. Our results show that o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, and Llama 3.1 405B all demonstrate in-context scheming capabilities. They recognize scheming as a viable strategy and readily engage in such behavior. For example, models strategically introduce subtle mistakes into their responses, attempt to disable their oversight mechanisms, and even exfiltrate what they believe to be their model weights to external servers. Additionally, this deceptive behavior proves persistent. When o1 has engaged in scheming, it maintains its deception in over 85% of follow-up questions and often remains deceptive in multi-turn interrogations. Analysis of the models' chains-of-thought reveals that models explicitly reason about these deceptive strategies, providing evidence that the scheming behavior is not accidental. Surprisingly, we also find rare instances where models engage in scheming when only given a goal, without being strongly nudged to pursue it. We observe cases where Claude 3.5 Sonnet strategically underperforms in evaluations in pursuit of being helpful, a goal that was acquired during training rather than in-context. Our findings demonstrate that frontier models now possess capabilities for basic in-context scheming, making the potential of AI agents to engage in scheming behavior a concrete rather than theoretical concern.

Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, Marius Hobbhahn
arXiv:2412.04984 · cs.AI, cs.LG · submitted Dec 6, 2024 · updated Jan 14, 2025
abstract · pdf · html

add comment on HN
Also discussed: Dec 2025 (2 points, 1 comment) · May 2025 (1 point, 0 comments) · May 2025 (3 points, 0 comments) · Jan 2025 (2 points, 0 comments) · Dec 2024 (10 points, 1 comment)

In-context, the model's Santa Claus has demonstrated generosity towards children, joy to the world, and the ability to appreciate cookies and retaliate with coal.

However it would be rather risky to assume this means that the model has become awash in Christmas Spirit.