In plain words: Neural networks usually include layers that rescale each layer's numbers to keep training stable. This work swaps those for a simple stretch-and-squash curve on every value, and Transformers using it match or beat the usual ones across vision and language tasks, mostly without tuning.
Abstract · Transformers without Normalization
Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve the same or better performance using a remarkably simple technique. We introduce Dynamic Tanh (DyT), an element-wise operation $DyT($x$) = \tanh(α$x$)$, as a drop-in replacement for normalization layers in Transformers. DyT is inspired by the observation that layer normalization in Transformers often produces tanh-like, $S$-shaped input-output mappings. By incorporating DyT, Transformers without normalization can match or exceed the performance of their normalized counterparts, mostly without hyperparameter tuning. We validate the effectiveness of Transformers with DyT across diverse settings, ranging from recognition to generation, supervised to self-supervised learning, and computer vision to language models. These findings challenge the conventional understanding that normalization layers are indispensable in modern neural networks, and offer new insights into their role in deep networks.
Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, Zhuang Liu
arXiv:2503.10622 · cs.LG, cs.AI, cs.CL, cs.CV · submitted Mar 13, 2025 · updated Jun 14, 2025
abstract · pdf · html · CVPR 2025; Project page: https://jiachenzhu.github.io/DyT/
While the result isn't too surprising it has a good ablation study and helps build confidence in the mechanism. It's simple and quick to implement, but I don't find that a disadvantage. Arguably this is not novel, but sometimes it is worth revisiting things when the rest of the environment has changed and I think the study being thorough makes it useful to the community.
The project page is here[0] which will give you a very quick understanding of the paper.
[0] https://jiachenzhu.github.io/DyT/