about
Predicting the Order of Upcoming Tokens Improves Language Modeling (arxiv.org)
7 points by wavelander on Aug 28, 2025 | hide | past | pdf | 2 comments on HN

In plain words: Instead of guessing which words come next, the model learns to rank upcoming words by how soon they appear, using one extra output layer instead of several extra layers. This beat next-word training and exact future-word guessing on nine benchmarks, even at 7 billion parameters.

Abstract

Multi-token prediction (MTP) has been proposed as an auxiliary objective to improve next-token prediction (NTP) in language model training but shows inconsistent improvements, underperforming in standard NLP benchmarks. We found MTP's exact future token prediction to be too difficult as an auxiliary loss. Instead, we propose token order prediction (TOP), which trains models to order upcoming tokens by their proximity using a learning-to-rank loss. TOP requires only a single additional unembedding layer compared to MTP's multiple transformer layers. We pretrain models of 340M, 1.8B, and 7B parameters using NTP, MTP, DeepSeek MTP (DS-MTP) and TOP objectives. The results of nine standard NLP benchmarks show that TOP overall outperforms NTP, MTP, and DS-MTP even at scale. TOP models with continued training on math and code also perform better on 4 relevant benchmarks. On the synthetic star graph task, TOP enables pathfinding on graphs where NTP, MTP, and DS-MTP fail. Our code is available at https://github.com/zaydzuhri/token-order-prediction

Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji
arXiv:2508.19228 · cs.LG · submitted Aug 26, 2025 · updated Feb 16, 2026
abstract · pdf · html

add comment on HN

Are any of these methods doable on pre-trained models? Like freeze the model and only train these add-ons? Having to redo the training runs with these optimisations doesn't sound too practical, in the great scheme of things.
It's obviously practical for the next model you train from scratch. The point of research is obviously not to improve existing commercial products.