In plain words: Examined 4,022 pull requests written by coding agents, checking every file change for security mistakes using an AI checker plus manual review. Nearly 39% contained at least one such mistake, and most leaked passwords slipped past both automated and human checks.
Abstract
The increasing adoption of autonomous coding agents accelerates software development but also introduces scoped security risks within high-impact file paths that can outpace traditional human review capacity. While prior research has primarily evaluated these systems in terms of functional correctness and productivity, this paper presents a large-scale empirical study using the AIDev dataset to systematically characterize security code smells in agent-generated pull requests (PRs). Through a combination of a validated LLM-as-a-judge framework and manual qualitative analysis, we identify and classify security misconfigurations across 16,112 file changes spanning 4,022 pull requests. Our results reveal that 38.9% of agent-generated PRs contain at least one security smell, with supply chain integrity issues accounting for 82.3% of all detected security smells. Furthermore, hard-coded credentials constitute 99.6% of all critical-severity security smells. Crucially, we find that human collaborators are responsible for introducing 67.6% of genuine leaked secrets within these agent-assisted workflows, while existing automated and human review processes fail to detect 81.1% of these credentials prior to integration. These findings highlight substantial security risks in agent-assisted software development workflows and suggest a potential reduction in developer vigilance. They also underscore the urgent need for context-aware security guardrails implemented directly at the point of human-AI collaboration.
A H M Nazmus Sakib, Dipayan Banik, Murtuza Jadliwala
arXiv:2607.12428 · cs.CR · submitted Jul 14, 2026 · updated Jul 19, 2026
abstract · pdf · html · Accepted at the KDD 2026 Workshop on Agentic Software Engineering (AgenticSE)