about
Recap: Reproducing Copyrighted Data from LLMs Training with an Agentic Pipeline (arxiv.org)
2 points by PaulHoule 317 days ago | hide | past | pdf | discuss on HN

In plain words: A tool coaxes a model into repeating copyrighted text it memorized: a second model checks attempts against the real passage and feeds back hints, and gets past refusals. On 30 full books, this lifted the text-match score from 0.38 to 0.47 versus one try.

Abstract · RECAP: Reproducing Copyrighted Data from LLMs Training with an Agentic Pipeline

If we cannot inspect the training data of a large language model (LLM), how can we ever know what it has seen? We believe the most compelling evidence arises when the model itself freely reproduces the target content. As such, we propose RECAP, an agentic pipeline designed to elicit and verify memorized training data from LLM outputs. At the heart of RECAP is a feedback-driven loop, where an initial extraction attempt is evaluated by a secondary language model, which compares the output against a reference passage and identifies discrepancies. These are then translated into minimal correction hints, which are fed back into the target model to guide subsequent generations. In addition, to address alignment-induced refusals, RECAP includes a jailbreaking module that detects and overcomes such barriers. We evaluate RECAP on EchoTrace, a new benchmark spanning over 30 full books, and the results show that RECAP leads to substantial gains over single-iteration approaches. For instance, with GPT-4.1, the average ROUGE-L score for the copyrighted text extraction improved from 0.38 to 0.47 - a nearly 24% increase.

André V. Duarte, Xuying li, Bin Zeng, Arlindo L. Oliveira, Lei Li, Zhuo Li
arXiv:2510.25941 · cs.CL · submitted Oct 29, 2025 · updated Mar 13, 2026
abstract · pdf · html

add comment on HN