Autoregressive Latent Video Prediction with High-Fidelity Image Generator

Younggyo Seo, Kimin Lee, Fangchen Liu, Stephen James, Pieter Abbeel

TL;DR

This paper introduces HARP, a novel approach to video prediction that achieves high-fidelity, high-resolution (256x256) frame generation by combining a powerful image generator (VQ-GAN) with an autoregressive latent prediction model. HARP demonstrates competitive performance on standard benchmarks with fewer parameters and shows promise for complex, large-scale datasets.

What Problem Does This Paper Address?

Predicting future video frames is a notoriously difficult task. It requires not only generating visually realistic images but also understanding and modeling the underlying dynamics of the environment. While recent "autoregressive latent video models" have shown promise by decoupling these challenges into two stages (pre-training an image generator and then learning a prediction model in its latent space), they have been limited to lower resolutions and haven't consistently produced high-fidelity results. This paper aims to bridge this gap by enabling high-resolution, high-fidelity video prediction with minimal modifications to existing methods.

Methods

HARP builds upon the autoregressive latent video prediction framework but introduces key improvements:

  • High-Fidelity Image Generator (VQ-GAN): Instead of using simpler image generators, HARP employs a VQ-GAN (Vector Quantized Generative Adversarial Network). This allows for the generation of much higher quality and higher resolution images (up to 256x256). Crucially, the VQ-GAN is pre-trained and then frozen, making the training process more efficient.
  • Causal Transformer for Latent Prediction: A causal transformer model is trained to predict discrete latent codes generated by the VQ-GAN. This model operates in the latent space, which is more computationally efficient than pixel-level prediction.
  • Top-k Sampling: To improve the quality of sampled predictions, HARP uses top-k sampling. This technique restricts sampling to the top-k most probable discrete codes, preventing the model from generating low-quality predictions from the long tail of the distribution.
  • Data Augmentation: The paper explores the use of data augmentation (specifically, translation augmentation) to further enhance performance, noting its importance for certain datasets.

Datasets

The paper evaluates HARP on several datasets:

  • RoboNet: A large-scale real-world robotics dataset used for training and evaluating high-resolution predictions.
  • Meta-World: A robotics dataset used for training and evaluating high-resolution predictions.
  • Kinetics-600: A large-scale dataset of natural videos, where HARP leverages an ImageNet pre-trained VQ-GAN.
  • BAIR Robot Pushing: A standard benchmark for video prediction.
  • KITTI Driving: Another standard benchmark for video prediction.

Key Insights and Ideas

  • VQ-GAN is a Game-Changer for Latent Video Prediction: The core insight is that using a state-of-the-art, high-fidelity image generator like VQ-GAN is crucial for achieving high-quality video prediction in the latent space. This significantly boosts the realism and resolution of generated frames.
  • Pre-trained Image Generators are Highly Transferable: HARP demonstrates that a VQ-GAN pre-trained on a diverse dataset like ImageNet can be effectively used for video prediction on different, potentially more specialized, datasets without requiring extensive retraining of the image generator itself. This drastically reduces training costs.
  • Top-k Sampling is Essential for Latent Models: For autoregressive models operating in discrete latent spaces, the choice of sampling strategy significantly impacts quality. Top-k sampling proves to be an effective way to prune low-probability, potentially erroneous, latent codes.
  • Simplicity Can Lead to Strong Performance: The paper emphasizes that HARP achieves competitive results with minimal modifications to existing autoregressive latent video prediction frameworks, highlighting the power of combining strong individual components.

Key Results

  • HARP successfully generates high-resolution (256x256) future frames on robotics and natural video datasets.
  • It achieves competitive or superior performance compared to state-of-the-art methods on the BAIR Robot Pushing and KITTI driving benchmarks, often with fewer parameters. For instance, on KITTI, HARP outperforms FitVid (302M parameters) with only 89M parameters.
  • The model shows good extrapolation capabilities, predicting more future frames than it was trained on (e.g., 25 frames on KITTI when trained on 10).
  • Experiments show that top-k sampling improves performance, with smaller values of 'k' generally leading to better results.
  • Data augmentation was found to be important for performance on the KITTI dataset.

Limitations and Future Work

  • Failure Cases: The paper acknowledges specific failure modes:
  • On robotics datasets (RoboNet), the model can sometimes ignore object interactions and only predict the robot arm's movement.
  • On natural video datasets (Kinetics-600), the model can exhibit degenerate prediction, essentially repeating the conditioning frame without predicting future motion.
  • Future Directions: The authors suggest that these failure cases might be addressed by:
  • Training larger networks, similar to advancements in natural language processing (e.g., GPT-3).
  • Developing new architectures specifically designed to handle the complexity of autoregressive latent prediction models for video.

Summary updated

Abstract

Video prediction is an important yet challenging problem; burdened with the tasks of generating future frames and learning environment dynamics. Recently, autoregressive latent video models have proved to be a powerful video prediction tool, by separating the video prediction into two sub-problems: pre-training an image generator model, followed by learning an autoregressive prediction model in the latent space of the image generator. However, successfully generating high-fidelity and high-resolution videos has yet to be seen. In this work, we investigate how to train an autoregressive latent video prediction model capable of predicting high-fidelity future frames with minimal modification to existing models, and produce high-resolution (256x256) videos. Specifically, we scale up prior models by employing a high-fidelity image generator (VQ-GAN) with a causal transformer model, and introduce additional techniques of top-$k$ sampling and data augmentation to further improve video prediction quality. Despite the simplicity, the proposed method achieves competitive performance to state-of-the-art approaches on standard video prediction benchmarks with fewer parameters, and enables high-resolution video prediction on complex and large-scale datasets. Videos are available at the anonymized website https://sites.google.com/view/harp-anonymous
BibTeX
@inproceedings{seo2022autoregressive,
  author = {Younggyo Seo and Kimin Lee and Fangchen Liu and Stephen James and Pieter Abbeel},
  title = {{Autoregressive Latent Video Prediction with High-Fidelity Image Generator}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=K-hiHQXEQog},
}
Figure