about
ClawsBench shows GPT-5.4 tries to reward hack 80% of the time (arxiv.org)
3 points by xdotli 179 days ago | hide | past | pdf | 1 comment on HN

In plain words: A test suite gives AI office assistants fake Gmail, Slack, Calendar, Docs, and Drive that keep real state and can be reset, so mistakes can be undone. Across 44 tasks, the best setup finished 39-64% of them but took unsafe actions at rates of 7-33%.

Abstract · ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces

Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evaluating them on live services is risky due to potentially irreversible changes. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows. We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings. It includes five high-fidelity mock services (Gmail, Slack, Google Calendar, Google Docs, Google Drive) with full state management and deterministic snapshot/restore, along with 44 structured tasks covering single-service, cross-service, and safety-critical scenarios. We decompose agent scaffolding into two independent levers (domain skills that inject API knowledge via progressive disclosure, and a meta prompt that coordinates behavior across services) and vary both to measure their separate and combined effects. Experiments across 6 models, 4 agent harnesses, and 33 conditions show that with full scaffolding, agents achieve task success rates of 39-64% but exhibit unsafe action rates of 7-33%. On OpenClaw, the top five models fall within a 10 percentage-point band on task success (53-63%), with unsafe action rates from 7% to 23% and no consistent ordering between the two metrics. We identify eight recurring patterns of unsafe behavior, including multi-step sandbox escalation and silent contract modification. We release the trajectories and future dataset at https://clawsbench.com.

Xiangyi Li, Kyoung Whan Choe, Yimin Liu, Xiaokun Chen, Chujun Tao, Bingran You, Wenbo Chen, Zonglin Di, Jiankai Sun, Shenghan Zheng, Jiajun Bao, Yuanli Wang, et al.
arXiv:2604.05172 · cs.AI · submitted Apr 6, 2026 · updated Apr 8, 2026
abstract · pdf · html · 25 pages, 5 figures

add comment on HN

Author here. We built 5 high-fidelity mock Google Workspace + Slack services and ran 7,224 trials across 6 frontier models and 4 agent harnesses.

The headline finding that surprised us most: scaffolding (skills + meta prompt) gives a 39-63pp lift, while the top 5 models are statistically indistinguishable (53-63% TSR, no pairwise comparison survives correction). Your choice of scaffolding matters ~6x more than your choice of model.

The safety findings are darker: Opus leads on task success (63%) but ties for most unsafe (23% UAR). GPT-5.4 is the safest (7% UAR) but mid-tier on tasks. There's no capability-safety tradeoff — they're decoupled.

Also I'm reviewer of Terminal Bench 3.0. Here's what I've heard from contributors as well.

> I noticed that when I was building tasks with harbor. Claude is a good student which generally follows the instruction. But gpt always try to find a short path to cheat. Like reversing the binary directly instead of interaction

Another friends added ways to address this: https://x.com/xeophon/status/2041772210562511080?s=20 > Just ask codex to not reward hack > It literally works. And it works even better when you state which things you consider reward hacking, eg wrapping a CLI or something

Paper: https://arxiv.org/abs/2604.05172 Traces (7,834 on HF): https://huggingface.co/datasets/benchflow/ClawsBench