about
Mixture-of-Recursions: Learning Dynamic Recursive Depths (arxiv.org)
1 point by nill0 on Jul 21, 2025 | hide | past | pdf | discuss on HN

In plain words: The same layers run repeatedly, with routers deciding how many passes each word needs, so only active words get attention and are stored. At equal training cost and smaller size, it predicted words more accurately and beat plain and recycled-layer models on few-shot tasks.

Abstract · Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation

Scaling language models unlocks impressive capabilities, but the accompanying computational and memory demands make both training and deployment expensive. Existing efficiency efforts typically target either parameter sharing or adaptive computation, leaving open the question of how to attain both simultaneously. We introduce Mixture-of-Recursions (MoR), a unified framework that combines the two axes of efficiency inside a single Recursive Transformer. MoR reuses a shared stack of layers across recursion steps to achieve parameter efficiency, while lightweight routers enable adaptive token-level thinking by dynamically assigning different recursion depths to individual tokens. This allows MoR to focus quadratic attention computation only among tokens still active at a given recursion depth, further improving memory access efficiency by selectively caching only their key-value pairs. Beyond these core mechanisms, we also propose a KV sharing variant that reuses KV pairs from the first recursion, specifically designed to further decrease memory footprint. Across model scales ranging from 135M to 1.7B parameters, MoR forms a new Pareto frontier: at equal training FLOPs and smaller model sizes, it significantly lowers validation perplexity and improves few-shot accuracy, while delivering higher throughput compared with vanilla and existing recursive baselines. These gains demonstrate that MoR is an effective path towards large-model quality without incurring large-model cost.

Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, Se-Young Yun
arXiv:2507.10524 · cs.CL, cs.LG · submitted Jul 14, 2025 · updated Oct 25, 2025
abstract · pdf · html · 38 pages, 9 figures, 17 tables, codes at https://github.com/raymin0223/mixture_of_recursions

add comment on HN
Also discussed: Jul 2025 (3 points, 0 comments) · Jul 2025 (1 point, 0 comments)