about
VideoFlow: A Flow-Based Generative Model for Video (arxiv.org)
13 points by iron0013 on Mar 5, 2019 | hide | past | pdf | 1 comment on HN

In plain words: Instead of predicting one fixed future or generating pixels one at a time, this model turns random noise into whole future videos while exactly tracking how likely each one is. It produced sharp, varied predictions and matched the best competing approaches.

Abstract · VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation

Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions. However, a central challenge in video prediction is that the future is highly uncertain: a sequence of past observations of events can imply many possible futures. Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally as in the case of pixel-level autoregressive models, or do not directly optimize the likelihood of the data. To our knowledge, our work is the first to propose multi-frame video prediction with normalizing flows, which allows for direct optimization of the data likelihood, and produces high-quality stochastic predictions. We describe an approach for modeling the latent space dynamics, and demonstrate that flow-based generative models offer a viable and competitive approach to generative modelling of video.

Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, Durk Kingma
arXiv:1903.01434 · cs.CV, cs.AI, cs.LG · submitted Mar 4, 2019 · updated Feb 12, 2020
abstract · pdf · html · ICLR 2020 Camera-Ready. Previous title: VideoFlow: A Flow-Based Generative Model for Video

add comment on HN

The paper references the code for the project which can be found here: https://github.com/tensorflow/tensor2tensor/blob/master/tens...