about
LCM-LoRA: A Universal Stable-Diffusion Acceleration Module (arxiv.org)
8 points by mromanuk on Nov 15, 2023 | hide | past | pdf | discuss on HN

In plain words: A small add-on is trained to make text-to-image generators draw good pictures in just a few steps, then can be dropped into any fine-tuned version without extra training. It worked across many image models using less memory and beat earlier shortcut samplers.

Abstract

Latent Consistency Models (LCMs) have achieved impressive performance in accelerating text-to-image generative tasks, producing high-quality images with minimal inference steps. LCMs are distilled from pre-trained latent diffusion models (LDMs), requiring only ~32 A100 GPU training hours. This report further extends LCMs' potential in two aspects: First, by applying LoRA distillation to Stable-Diffusion models including SD-V1.5, SSD-1B, and SDXL, we have expanded LCM's scope to larger models with significantly less memory consumption, achieving superior image generation quality. Second, we identify the LoRA parameters obtained through LCM distillation as a universal Stable-Diffusion acceleration module, named LCM-LoRA. LCM-LoRA can be directly plugged into various Stable-Diffusion fine-tuned models or LoRAs without training, thus representing a universally applicable accelerator for diverse image generation tasks. Compared with previous numerical PF-ODE solvers such as DDIM, DPM-Solver, LCM-LoRA can be viewed as a plug-in neural PF-ODE solver that possesses strong generalization abilities. Project page: https://github.com/luosiallen/latent-consistency-model.

Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, Hang Zhao
arXiv:2311.05556 · cs.CV, cs.LG · submitted Nov 9, 2023
abstract · pdf · html · Technical Report

add comment on HN
Also discussed: Nov 2023 (4 points, 0 comments)