In plain words: Training image-generating transformers at many sizes and data amounts, they found the loss falls along a predictable curve as computing power grows. The curve lets them choose the best model size and data for a budget and predict quality, since loss tracked image quality across datasets.
Abstract · Scaling Laws For Diffusion Transformers
Diffusion transformers (DiT) have already achieved appealing synthesis and scaling properties in content recreation, e.g., image and video generation. However, scaling laws of DiT are less explored, which usually offer precise predictions regarding optimal model size and data requirements given a specific compute budget. Therefore, experiments across a broad range of compute budgets, from 1e17 to 6e18 FLOPs are conducted to confirm the existence of scaling laws in DiT for the first time. Concretely, the loss of pretraining DiT also follows a power-law relationship with the involved compute. Based on the scaling law, we can not only determine the optimal model size and required data but also accurately predict the text-to-image generation loss given a model with 1B parameters and a compute budget of 1e21 FLOPs. Additionally, we also demonstrate that the trend of pre-training loss matches the generation performances (e.g., FID), even across various datasets, which complements the mapping from compute to synthesis quality and thus provides a predictable benchmark that assesses model performance and data quality at a reduced cost.
Zhengyang Liang, Hao He, Ceyuan Yang, Bo Dai
arXiv:2410.08184 · cs.CV · submitted Oct 10, 2024 · updated Mar 4, 2026
abstract · pdf · html