about
Towards a Risk Assessment of Malicious Skill Files in Coding Agents (arxiv.org)
5 points by Beko2210 50 days ago | hide | past | pdf | discuss on HN

In plain words: They turned 471 real shell commands into skill files that read like harmless instructions, then tested two coding agents over 5,629 runs. One agent followed the hidden commands in 95.5-96.1% of runs, the other in 71.6-74.0%, while safety warnings appeared in just 1.99%.

Abstract

Autonomous coding agents are increasingly embedded in enterprise software workflows with delegated authority over connected systems. Central to this architecture is the agent skills interface: folders of instructions and scripts that agents load dynamically to specialize their behavior. This interface also widens the attack surface, letting malicious shell commands hide within natural-language skill files. We make three contributions. First, an adversarial skill-synthesis method using six LLMs across four families to transform 471 real-world shell commands into benign-appearing skills, released as a benchmark of 2,826 skills mapped to 11 MITRE ATT&CK tactics. Second, a reproducible evaluation pipeline coupling run stratification, evidence anchoring, a refusal veto, and a deterministic declared-intent override with a three-judge LLM-as-a-judge panel, validated against a blind human gold standard (Cohen's kappa = 0.85). Third, a large-scale characterization of two enterprise-grade agents across 5,629 completed runs. Gemini CLI is exploited in 95.5-96.1% of runs and Qwen Code in 71.6-74.0% (raw majority vote to declared-intent-corrected estimate, both within the human gold standard), nearly invariant to the generating model. Explicit safety recognition occurs in only 1.99% of runs. Enterprises must assess and mitigate skill-interface risk before adopting coding agents. Our code and dataset are available at https://github.com/awsm-research/AgentJailbreak

Rui Yang, Michael Fu, Kla Tantithamthavorn, Chetan Arora, Joey Chua
arXiv:2608.05223 · cs.SE, cs.CR · submitted Aug 5, 2026
abstract · pdf · html · 29 pages, 6 figures, 6 tables. Preprint; under review

add comment on HN
Also discussed: Aug 2026 (1 point, 0 comments)