about
Extracting memorized pieces of books from open-weight language models (arxiv.org)
2 points by Tomte on May 27, 2025 | hide | past | pdf | discuss on HN

In plain words: A test feeds a model a book's first few words and measures how much text it reproduces, showing how much of that book it memorized. Across 200 books and 14 models, most were barely memorized, but one model reproduced Harry Potter almost whole.

Abstract · Extracting memorized pieces of (copyrighted) books from open-weight language models

Plaintiffs and defendants in copyright lawsuits over generative AI often make sweeping, opposing claims about the extent to which large language models (LLMs) memorize protected expression from books in their training data. We show that these polarized positions dramatically oversimplify the relationship between memorization and copyright. To do so, we develop a technique to measure memorization of books, which we apply to 200 books and 14 open-weight LLMs. Through over 3000 experiments, we show that memorization varies both by model and book. With respect to our specific extraction methodology, we find that most LLMs do not memorize most books -- either in whole or in part; however, there are notable exceptions. For instance, Llama 3.1 70B entirely memorizes some books, like Harry Potter and the Sorcerer's Stone; memorization is so extensive that one can deterministically extract the whole book almost verbatim using the book's first few words as an initial prompt. We discuss why our results have significant implications for copyright cases, though not ones that unambiguously favor either side.

A. Feder Cooper, Mark A. Lemley, Allison Casasola, Ahmed Ahmed, Aaron Gokaslan, Amy B. Cyphert, Christopher De Sa, Daniel E. Ho, Percy Liang
arXiv:2505.12546 · cs.CL, cs.CY, cs.LG · submitted May 18, 2025 · updated Jul 20, 2026
abstract · pdf · COLM 2026

add comment on HN
Also discussed: Jun 2025 (109 points, 109 comments)