about
Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation (arxiv.org)
1 point by sinnsro 36 days ago | hide | past | pdf | discuss on HN

In plain words: Instead of chopping text into word pieces first, this model reads raw bytes and, with special training tricks, learns its own word-like boundaries. As it grows it beats the usual word-piece models, and guessing ahead at those boundaries accepts 3.4 times more tokens.

Abstract

Tokenizer-free language models remove the inductive bias of fixed tokenizers by modeling text directly as bytes, but the resulting longer sequences substantially increase computation and eliminate explicit text abstractions. We ask whether this additional computation can be useful, and whether standard Transformers can learn the abstractions that tokenization provides. We study these questions on Transformers without specialized tokenization-related architectures. With token-superposition training and hash embeddings, byte Transformers consistently outperform subword Transformers as model size scales. We further find that byte Transformers build local text abstractions as external tokenizers: a set of segmentation-like positions are used to collect local context representations, and restricting up to $25\%$ of intermediate layers to these local representations preserves downstream performance. Finally, these learned structures induce highly non-uniform generation difficulty, with uncertainty concentrated near local structure boundaries; exploiting them for speculative decoding yields $3.4\times$ more accepted tokens than in subword Transformers.

Jie Wang, Shiwei Luo, Qi Zhang, Yuanbin Wu
arXiv:2610.05978 · cs.CL · submitted Oct 5, 2026
abstract · pdf · html

add comment on HN