about
AI Agents vs. Pentesters (arxiv.org)
2 points by takira 293 days ago | hide | past | pdf | discuss on HN

In plain words: A new AI hacking team writes its own prompts, spawns agents, and checks whether found flaws are real; it faced human experts on a university network. It beat nine of the ten professionals, but raised more false alarms and struggled with on-screen controls.

Abstract · Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing

We present the first comprehensive evaluation of AI agents against human cybersecurity professionals in a live enterprise environment. We evaluate ten cybersecurity professionals alongside six existing AI agents and ARTEMIS, our new agent scaffold, on a large university network consisting of ~8,000 hosts across 12 subnets. ARTEMIS is a multi-agent framework featuring dynamic prompt generation, arbitrary sub-agents, and automatic vulnerability triaging. In our comparative study, ARTEMIS placed second overall, discovering 9 valid vulnerabilities with an 82% valid submission rate and outperforming 9 of 10 human participants. While existing scaffolds such as Codex and CyAgent underperformed relative to most human participants, ARTEMIS demonstrated technical sophistication and submission quality comparable to the strongest participants. We observe that AI agents offer advantages in systematic enumeration, parallel exploitation, and cost -- certain ARTEMIS variants cost $18/hour versus $60/hour for professional penetration testers. We also identify key capability gaps: AI agents exhibit higher false-positive rates and struggle with GUI-based tasks.

Justin W. Lin, Eliot Krzysztof Jones, Donovan Julian Jasper, Ethan Jun-shen Ho, Anna Wu, Arnold Tianyi Yang, Neil Perry, Andy Zou, Matt Fredrikson, J. Zico Kolter, Percy Liang, Dan Boneh, et al.
arXiv:2512.09882 · cs.AI, cs.CR, cs.CY · submitted Dec 10, 2025 · updated Mar 3, 2026
abstract · pdf · html

add comment on HN
Also discussed: Jan 2026 (125 points, 92 comments) · Dec 2025 (2 points, 0 comments)