about
A Fix for Softmax and Cross-Entropy (arxiv.org)
4 points by vpj 24 days ago | hide | past | pdf | discuss on HN

In plain words: Softmax and cross-entropy code has rounding errors that erase a needed gradient or invent one when it should be zero; rewriting the math fixes both. The fix removes training jumps and accuracy crashes, and shows output values stop growing once weight decay balances Adam's epsilon.

Abstract

Standard implementations of softmax, log-softmax, and cross-entropy have two numerical bugs. The first bug, Softmax Collapse, rounds the target logit's gradient to zero if that one logit is much larger than the others, resulting in gradients that no longer sum to zero. The second bug, Softmax Cancellation, occurs when student logits are close to teacher logits, and produces non-zero gradients even when the student and teacher logits are equal. We present simple implementations of log-softmax, softmax, and cross-entropy that fix both bugs. Our fix eliminates the previously-observed "Slingshot" phenomenon, speeds up grokking with weight decay, prevents accuracy collapse without weight decay, stops student networks from drifting away from the teacher during distillation, and shows that after memorization, logit growth is bounded by an equilibrium between weight decay and Adam's $ε$ hyperparameter.

Jonathan T. Barron, Dan Gnanapragasam, Michael Tschannen
arXiv:2610.31642 · cs.LG · submitted Oct 22, 2026
abstract · pdf · html

add comment on HN
Also discussed: Oct 2026 (4 points, 0 comments)