In plain words: A network reworks computer-generated images to look like real photos while keeping the simulator's labels, guided by unlabeled real photos. Gaze and hand-pose models trained on these polished images beat ones trained on synthetic images, setting a record on MPIIGaze without labeled real data.
Abstract
With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulator's output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a 'self-regularization' term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, Russ Webb
arXiv:1612.07828 · cs.CV, cs.LG, cs.NE · submitted Dec 22, 2016 · updated Jul 19, 2017
abstract · pdf · html · Accepted at CVPR 2017 for oral presentation
- Labeled data is very expensive. Historically attempts to learn on synthetic data has failed because ConvNets are very good at detecting small visual artifacts in the synthetic data and using those for classification during training. At test time on real data, those artifacts aren't present so model fails. A technique that can beat state-of-the-art (admittedly on a very narrow Eye Gaze dataset, but still) by only training on labels from synthetic data and testing on real data is important.
- They present a useful new idea to improve GAN training: using a history of "fake" images, rather than only the latest fake images from the generator. Ask anyone who has tried to train a GAN: the training is really unstable, each network only cares about beating the latest version of its "opponent". They show good improvements by saving many previous fake outputs to make the generator more robust. This reminds me of Experience Replay from DeepMind for RL.
- It's a published paper from Apple! Great that they are starting to contribute back to the research community.