 |
ML Newsnew | past | best | rss | submit | about |
| Transformer learning explained: Coinductive guide to inductive transformer heads (arxiv.org) | |
| 1 point by adamnemecek on Feb 27, 2023 | hide | past | pdf | 1 comment on HN | |
|
In plain words: Every transformer part fits one algebraic structure where attention acts like a smoothing operation and the residual stream like an echo. Because it carries its own gradient, layers learn by matching two paths inside themselves, with no backward pass across the network, unlike usual training. Abstract · Coinductive guide to inductive transformer headsWe argue that all building blocks of transformer models can be expressed with a single concept: combinatorial Hopf algebra. Transformer learning emerges as a result of the subtle interplay between the algebraic and coalgebraic operations of the combinatorial Hopf algebra. Viewed through this lens, the transformer model becomes a linear time-invariant system where the attention mechanism computes a generalized convolution transform and the residual stream serves as a unit impulse. Attention-only transformers then learn by enforcing an invariant between these two paths. We call this invariant Hopf coherence. Due to this, with a degree of poetic license, one could call combinatorial Hopf algebras "tensors with a built-in loss function gradient". This loss function gradient occurs within the single layers and no backward pass is needed. This is in contrast to automatic differentiation which happens across the whole graph and needs a explicit backward pass. This property is the result of the fact that combinatorial Hopf algebras have the surprising property of calculating eigenvalues by repeated squaring. | |
| add comment on HN |
I have recently made the realization that Hopf algebras are a surprisingly good abstraction for machine learning and this paper is an invitation for others to look into them as they can provide a unified foundation for convnets, transformers and diffusion models.
Hopf algebras are a tensorial bialgebra, meaning they are both a tensor and a cotensor at once. What's the co- prefix? You can think of it as "a thing with recurrence relationships". This matters since recurrence relationships can be thought of as a generalization of autodiff. The two ML building blocks, tensors and autodiff, are then unified under one umbrella.
Looking at transformers through the lens of Hopf algebra allowed me to identify their learning mechanism. Namely, the residual stream corresponds to what my paper calls unit impulse path and the attention corresponds to the convolution path of a Hopf algebra. The transformer learns by enforcing an invariant called Hopf coherence between the two paths.
Hopf convolution is a generalization of the standard convolution and transformer attention emerges as a part of this convolution. Transformer models can then be understood as linear time-invariant systems.
Furthermore, I believe that Hopf coherence provides a better way of doing backprop, one that occurs within the single layers as opposed to across the whole graph, but more research is needed.
This approach also opens the door for verified machine learning, which I will discuss in future work.
I'm currently in the process of implementing a simple machine learning framework based on Hopf algebra but it might take some time.