about
Neural Thermodynamic Laws for Large Language Model Training (arxiv.org)
6 points by anticensor on May 25, 2025 | hide | past | pdf | discuss on HN

In plain words: Unlike scaling laws, which just track how error shrinks as models grow, this treats training like heat moving through matter: temperature, entropy, and thermodynamic laws emerge when the error landscape looks like a river valley. Those rules give simple advice for setting learning speed.

Abstract

Beyond neural scaling laws, little is known about the laws underlying large language models (LLMs). We introduce Neural Thermodynamic Laws (NTL) -- a new framework that offers fresh insights into LLM training dynamics. On the theoretical side, we demonstrate that key thermodynamic quantities (e.g., temperature, entropy, heat capacity, thermal conduction) and classical thermodynamic principles (e.g., the three laws of thermodynamics and the equipartition theorem) naturally emerge under river-valley loss landscape assumptions. On the practical side, this scientific perspective yields intuitive guidelines for designing learning rate schedules.

Ziming Liu, Yizhou Liu, Jeff Gore, Max Tegmark
arXiv:2505.10559 · cs.LG, cs.AI, physics.data-an, stat.ML · submitted May 15, 2025
abstract · pdf · html · 18 pages, 10 figures

add comment on HN