about
Sir-Bench – benchmark for security incident response agents (arxiv.org)
6 points by dan_l2 171 days ago | hide | past | pdf | 2 comments on HN

In plain words: A test suite of 794 incidents replays real attacks in cloud sandboxes to check whether an AI agent digs up new evidence instead of just repeating the alert. The team's own agent averaged 5.67 fresh findings per case.

Abstract · SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents

We present SIR-Bench, a benchmark of 794 test cases for evaluating autonomous security incident response agents that distinguishes genuine forensic investigation from alert parroting. Derived from 129 anonymized incident patterns with expert-validated ground truth, SIR-Bench measures not only whether agents reach correct triage decisions, but whether they discover novel evidence through active investigation. To construct SIR-Bench, we develop Once Upon A Threat (OUAT), a framework that replays real incident patterns in controlled cloud environments, producing authentic telemetry with measurable investigation outcomes. Our evaluation methodology introduces three complementary metrics: triage accuracy (M1), novel finding discovery (M2), and tool usage appropriateness (M3), assessed through an adversarial LLM-as-Judge that inverts the burden of proof -- requiring concrete forensic evidence to credit investigations. Evaluating our SIR agent on the benchmark demonstrates 97.1% true positive (TP) detection, 73.4% false positive (FP) rejection, and 5.67 novel key findings per case, establishing a baseline against which future investigation agents can be measured.

Daniel Begimher, Cristian Leo, Jack Huang, Pat Gaw, Bonan Zheng
arXiv:2604.12040 · cs.CR, cs.AI, cs.SE · submitted Apr 13, 2026
abstract · pdf · html · 9 pages, 6 tables, 1 figure. Equal contribution by first three authors

add comment on HN

Authors here. Existing security-AI benchmarks measure knowledge, offensive capability, or workflow completion. SIR-Bench measures whether an agent discovers novel evidenceduring an investigation vs. reaching the right conclusion by restating the alert. 794 test cases derived from 129 real incident patterns, replayed in live cloud environments. Happy to answer questions on methodology, scoring, or failure modes we saw acrossfrontier models.
woah this is cool