about
Decision Models as Policy Gates for Coding Agents (arxiv.org)
2 points by ghshephard 14 days ago | hide | past | pdf | discuss on HN

In plain words: A decision model checks each tool call a coding agent is about to make against safety rules, spelled out with examples of allowed and blocked actions. This raised detection by 5.5 to 26 percentage points across 23 models and cut false alarms on 21.

Abstract · Decision Models as Policy Gates for Coding Agents: Explicit Policy Boundaries Improve Detection and Reduce False Alarms

Coding agents can destroy data, leak secrets and message the wrong person faster than a human can review. Could a decision model catch these risks at runtime? We evaluated 22 hosted decision models and two prompted general models as classifiers for a multi-rule policy engine. A synthetic dataset of 2,643 tool calls was labeled blind against 28 rules, with a sealed holdout. With a single shared preamble in both conditions, explicit boundaries written as examples and counterexamples raised recall by 5.5-26.1 percentage points on all 23 available models and reduced false positives on 21. Jev 1.14's recall rose from 89.3% to 98.2%, while unnecessary interruptions fell from 30.6% to 4.8% of 252 clean actions. With examples, its median latency was 120 ms at $0.46 per thousand requests. Strong models retained the recall gains when concrete names were replaced by generic descriptions; three largely retained them when the same boundaries were written as prose. On 16 new scenarios written after the initial studies, a follow-up improved recall on 22 of 23 models and reduced unnecessary interruptions on 21. We release the dataset, rule texts and harness.

James P. Beno
arXiv:2611.02155 · cs.CL, cs.CR, cs.SE · submitted Nov 1, 2026
abstract · pdf · html · 15 pages, 2 figures. Dataset, rule texts and harness available at https://github.com/agentapprove/decision-policy-rules

add comment on HN