about
Multi-Token Prediction Done Right (arxiv.org)
13 points by matt_d 13 days ago | hide | past | pdf | 4 comments on HN

In plain words: Teaching a model to guess several future words at once mixes a useful signal with a noisy one that drowns it out. A one-line change keeps only the useful part, lowering errors and speeding up fast text generation at every size tested.

Abstract

Multi-Token Prediction (MTP) is now a common ingredient in language-model pre-training, as it improves downstream performance and enables faster inference through speculative decoding. We identify a flaw in standard MTP training, in which a residual component of the MTP gradient interferes with both next-token and multi-token learning, increasingly so with scale. Specifically, the MTP cross-entropy gradient can be decomposed exactly into a KL self-distillation term that pulls each MTP head toward the model's own next-token prediction (NTP) distribution and improves speculative decoding, and a residual that has a low signal-to-noise ratio and buries the useful KL signal. We introduce the KL gradient-recovery trick: when two distributions share a linear output head, the gradient of their KL divergence can be recovered exactly by combining their cross-entropy gradients through straight-through gradient routing. Unlike minimizing the KL directly, this trick requires no vocabulary-sized logit matrices and is compatible with standard fused cross-entropy kernels. This enables Soft Multi-Token Prediction (Soft-MTP), a one-line code change to MTP that removes the noisy residual and backpropagates only the KL self-distillation signal by reusing the existing NTP and MTP cross-entropy computations. Soft-MTP supports both DeepSeek- and Meta-style MTP, adds no parameters, and incurs no measurable overhead. From 167M to 1.2B parameters, Soft-MTP improves over standard MTP with lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy at every scale, lower MTP perplexity from 689M onward, and stronger math and code reasoning after fine-tuning at 1.2B. Soft-MTP also matches, if not exceeds, direct KL divergence minimization on most metrics. Soft-MTP is, in effect, a free lunch for MTP optimization. Our code can be found at https://github.com/JaydenTeoh/Soft-MTP.

Jayden Teoh
arXiv:2610.38331 · cs.LG · submitted Oct 28, 2026
abstract · pdf · html · Preprint

add comment on HN

This is such an elegant paper. At the risk of minimizing the thought that went into it, I love how it boils down to cleverly introducing zero, twice (to decompose the hard MTP gradient and to show the effect of soft MTP on the gradient's bias and variance).

I wish I could come up with this kind of stuff, haha. Clearly came from spending some time looking carefully at and thinking carefully about MTP heads. Kudos and thanks for an interesting paper!

thank you for this comment! it was really a serendipitous idea—i was playing around with amortizing the KL computation in our previous Next-Latent Prediction work (https://arxiv.org/abs/2511.05963) and this Soft-MTP idea trickled out from it. i sincerely appreciate your kind words, it means a lot to me!
The results look like a statistical fluke to me. Tiny effect, sometimes negative.
hey thanks for the comment, author here. That's a fair read of the loss numbers, I could only fund up to 1.2B scale myself and I don't want to oversell the results. But note that the benefits that Soft-MTP delivers in the paper is *consistently growing across scale*. The residual is irreducible with scale and will only become noisier as model grows bigger, so removing it via Soft-MTP is intuitive. It is also easily verifiable cos it only needs minor code change, codebase: https://github.com/JaydenTeoh/Soft-MTP

there is reproduction of Soft-MTP's benefit over standard MTP at 35B scale but it is only one data point for now: https://x.com/HivazeMe/status/2118432486724391217