about
Copy-Paste Language Models (arxiv.org)
3 points by azhenley 31 days ago | hide | past | pdf | discuss on HN

In plain words: A small extra part of the model is trained to copy text straight from the input, alongside the usual job of predicting the next word. It learned faster and did better on tasks at every size tested, with almost no extra cost.

Abstract

Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.

Nathan Godey, Yoav Artzi
arXiv:2610.19247 · cs.CL, cs.AI · submitted Oct 13, 2026
abstract · pdf · html

add comment on HN