about
TDAD – Compiling Tool-Using Agents from Behavioral Specifications (arxiv.org)
2 points by tzafrir 202 days ago | hide | past | pdf | 1 comment on HN

In plain words: Engineers write rules for how an AI assistant should act; one agent turns them into tests, another rewrites the prompt until it passes, and hidden tests catch cheats. Across 24 trials, 92% of prompts succeeded on the first try, passing 97% of unseen tests.

Abstract · Test-Driven AI Agent Definition (TDAD): Compiling Tool-Using Agents from Behavioral Specifications

We present Test-Driven AI Agent Definition (TDAD), a methodology that treats agent prompts as compiled artifacts: engineers provide behavioral specifications, a coding agent converts them into executable tests, and a second coding agent iteratively refines the prompt until tests pass. Deploying tool-using LLM agents in production requires measurable behavioral compliance that current development practices cannot provide. Small prompt changes cause silent regressions, tool misuse goes undetected, and policy violations emerge only after deployment. To mitigate specification gaming, TDAD introduces three mechanisms: (1) visible/hidden test splits that withhold evaluation tests during compilation, (2) semantic mutation testing via a post-compilation agent that generates plausible faulty prompt variants, with the harness measuring whether the test suite detects them, and (3) spec evolution scenarios that quantify regression safety when requirements change. We evaluate TDAD on SpecSuite-Core, a benchmark of four deeply-specified agents spanning policy compliance, grounded analytics, runbook adherence, and deterministic enforcement. Across 24 independent trials, TDAD achieves 92% v1 compilation success with 97% mean hidden pass rate; evolved specifications compile at 58%, with most failed runs passing all visible tests except 1-2, and show 86-100% mutation scores, 78% v2 hidden pass rate, and 97% regression safety scores. The implementation is available as an open benchmark at https://github.com/f-labs-io/tdad-paper-code.

Tzafrir Rehan
arXiv:2603.08806 · cs.SE, cs.AI · submitted Mar 9, 2026
abstract · pdf · html · 9 pages, 2 figures, open benchmark at https://github.com/f-labs-io/tdad-paper-code

add comment on HN

I built a methodology at Fiverr Labs for generating agent prompts from product specs using tests instead of manual prompt engineering. You write a behavioral spec, a coding agent generates tests from it, and a second agent iterates on the prompt until tests pass. Hidden test splits and mutation testing address specification gaming.

Evaluated on 4 agent specs across 24 trials — 92% compilation success, $2–3 per compilation. The benchmark and all code are open at https://github.com/f-labs-io/tdad-paper-code

Happy to discuss the methodology, limitations, and directions for follow-ups