In plain words: A stack of small generators trains on one photo, each learning patterns at a different scale, to build new images from noise. Unlike earlier single-image tricks limited to textures, it makes varied pictures of any size keeping layout and detail, often mistaken for real photos.
Abstract
We introduce SinGAN, an unconditional generative model that can be learned from a single natural image. Our model is trained to capture the internal distribution of patches within the image, and is then able to generate high quality, diverse samples that carry the same visual content as the image. SinGAN contains a pyramid of fully convolutional GANs, each responsible for learning the patch distribution at a different scale of the image. This allows generating new samples of arbitrary size and aspect ratio, that have significant variability, yet maintain both the global structure and the fine textures of the training image. In contrast to previous single image GAN schemes, our approach is not limited to texture images, and is not conditional (i.e. it generates samples from noise). User studies confirm that the generated samples are commonly confused to be real images. We illustrate the utility of SinGAN in a wide range of image manipulation tasks.
Tamar Rott Shaham, Tali Dekel, Tomer Michaeli
arXiv:1905.01164 · cs.CV · submitted May 2, 2019 · updated Sep 4, 2019
abstract · pdf · html · ICCV 2019
On a 1080 Ti it takes 45-90 minutes to train networks for the various tasks on 256px images (depending on some quality parameters and which task). Each task also requires training individually so if you'd like to try them all for a given image you'll need to train 6 times.
Also the pyramid of GANs approach is very memory hungry. I was only able to get up to 724px images with 11 GB of VRAM. This was also only possible with a higher scale factor (sparser pyramid) which sacrifices a lot of quality and is incredibly noticeable at larger image sizes. I only tried for larger sizes with the animation task though, perhaps there is a way to combine the super resolution and animation task and achieve better results. Training on larger sizes was taking upwards of 6-8 hours.
All of this was tested with the official repo[1] about a month ago.
[1] https://github.com/tamarott/SinGAN