about
Visual Autoregressive Modeling: Image Generation via Next-Resolution Prediction (arxiv.org)
1 point by famouswaffles on Oct 5, 2024 | hide | past | pdf | 1 comment on HN

In plain words: Instead of filling in an image pixel by pixel in reading order, this generates it in whole steps from blurry to detailed. On ImageNet 256x256, that cut the usual image-quality error score from 18.65 to 1.73 and made images about 20 times faster.

Abstract · Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction

We present Visual AutoRegressive modeling (VAR), a new generation paradigm that redefines the autoregressive learning on images as coarse-to-fine "next-scale prediction" or "next-resolution prediction", diverging from the standard raster-scan "next-token prediction". This simple, intuitive methodology allows autoregressive (AR) transformers to learn visual distributions fast and generalize well: VAR, for the first time, makes GPT-like AR models surpass diffusion transformers in image generation. On ImageNet 256x256 benchmark, VAR significantly improve AR baseline by improving Frechet inception distance (FID) from 18.65 to 1.73, inception score (IS) from 80.4 to 350.2, with around 20x faster inference speed. It is also empirically verified that VAR outperforms the Diffusion Transformer (DiT) in multiple dimensions including image quality, inference speed, data efficiency, and scalability. Scaling up VAR models exhibits clear power-law scaling laws similar to those observed in LLMs, with linear correlation coefficients near -0.998 as solid evidence. VAR further showcases zero-shot generalization ability in downstream tasks including image in-painting, out-painting, and editing. These results suggest VAR has initially emulated the two important properties of LLMs: Scaling Laws and zero-shot task generalization. We have released all models and codes to promote the exploration of AR/VAR models for visual generation and unified learning.

Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, Liwei Wang
arXiv:2404.02905 · cs.CV, cs.AI · submitted Apr 3, 2024 · updated Jun 10, 2024
abstract · pdf · html · Demo website: https://var.vision/

add comment on HN

Autoregressive Transformers scale and learn much better/faster when trained to predict the "next-resolution"/"next-scale", i.e start very small and gradually scale up the resolution and size vs being trained to predict the next image token/patch.

Related: VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling - https://arxiv.org/abs/2408.01181

STAR: Scale-wise Text-to-image generation via Auto-Regressive representations https://arxiv.org/abs/2406.10797