In plain words: It blurs your image or edit with noise, then lets an image generator clean it into a realistic photo; the noise level trades realism against staying true to your input. Human raters scored it up to 98% higher on realism than older single-pass generators.
Abstract · SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
Guided image synthesis enables everyday users to create and edit photo-realistic images with minimum effort. The key challenge is balancing faithfulness to the user input (e.g., hand-drawn colored strokes) and realism of the synthesized image. Existing GAN-based methods attempt to achieve such balance using either conditional GANs or GAN inversions, which are challenging and often require additional training data or loss functions for individual applications. To address these issues, we introduce a new image synthesis and editing method, Stochastic Differential Editing (SDEdit), based on a diffusion model generative prior, which synthesizes realistic images by iteratively denoising through a stochastic differential equation (SDE). Given an input image with user guide of any type, SDEdit first adds noise to the input, then subsequently denoises the resulting image through the SDE prior to increase its realism. SDEdit does not require task-specific training or inversions and can naturally achieve the balance between realism and faithfulness. SDEdit significantly outperforms state-of-the-art GAN-based methods by up to 98.09% on realism and 91.72% on overall satisfaction scores, according to a human perception study, on multiple tasks, including stroke-based image synthesis and editing as well as image compositing.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon
arXiv:2108.01073 · cs.CV, cs.AI · submitted Aug 2, 2021 · updated Jan 5, 2022
abstract · pdf · html · https://sde-image-editing.github.io/
Backward SDEs aren't time-reversed, but satisfy conditions at the end of the time interval instead of the beginning. The idea of time-reversing Brownian motion is like saying that you're running thermodynamics backward--it only makes sense if you sample them forward, then move backwards along the forward sampled motion.
It feels like (2) is just a backward SDE arrived at via the Feynman-Kac theorem applied to the Kolmogorov backward equations.