about
The statistical thermodynamics of generative diffusion models (arxiv.org)
1 point by alexmolas on Nov 13, 2023 | hide | past | pdf | discuss on HN

In plain words: Rewriting how diffusion models generate data using the physics of systems at thermal balance reveals sharp phase changes where one pattern wins out. These transitions always follow simple average-field behavior, and the instability at the changeover is what gives the models their power to create.

Abstract · The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability

Generative diffusion models have achieved spectacular performance in many areas of machine learning and generative modeling. While the fundamental ideas behind these models come from non-equilibrium physics, variational inference and stochastic calculus, in this paper we show that many aspects of these models can be understood using the tools of equilibrium statistical mechanics. Using this reformulation, we show that generative diffusion models undergo second-order phase transitions corresponding to symmetry breaking phenomena. We show that these phase-transitions are always in a mean-field universality class, as they are the result of a self-consistency condition in the generative dynamics. We argue that the critical instability that arises from the phase transitions lies at the heart of their generative capabilities, which are characterized by a set of mean-field critical exponents. Finally, we show that the dynamic equation of the generative process can be interpreted as a stochastic adiabatic transformation that minimizes the free energy while keeping the system in thermal equilibrium.

Luca Ambrogioni
arXiv:2310.17467 · stat.ML, cs.LG · submitted Oct 26, 2023 · updated Jun 20, 2024
abstract · pdf · html

add comment on HN
Also discussed: Oct 2023 (3 points, 0 comments)