Abstract
Video prediction is an important yet challenging problem, burdened with the dual tasks of generating future frames and learning environment dynamics. Recently, autoregressive latent video models have emerged as a powerful tool by separating video prediction into two sub-problems: pre-training an image generator model, followed by learning an autoregressive prediction model in the latent space. However, successfully generating high-fidelity and high-resolution videos remains difficult. In this work, we investigate how to train an autoregressive latent video prediction model capable of predicting high-fidelity future frames with minimal modification to existing models, producing high-resolution (256x256) videos. Specifically, we scale up prior models by employing a high-fidelity image generator (VQ-GAN) with a causal transformer model and introduce additional techniques such as top-k sampling and data augmentation to improve prediction quality. Despite its simplicity, the proposed method, HARP, achieves competitive performance to state-of-the-art approaches on standard benchmarks with fewer parameters and enables high-resolution video prediction on complex, large-scale datasets.
Introduction
Video prediction is often described as a "two-headed monster." To succeed, a model must simultaneously master two distinct domains: high-fidelity image synthesis and the complex underlying physics of environmental dynamics. Traditional models that attempt to solve both tasks simultaneously in pixel space often face significant computational hurdles. As resolution increases, these models frequently produce blurry or "soupy" results because modeling every pixel's transition is extremely high-dimensional.
While latent models—which predict future states in a compressed representation—have shown promise, they have historically been restricted to lower resolutions like 64x64. HARP (High-fidelity AutoRegressive latent video Prediction) demonstrates that a modular approach, decoupling high-fidelity reconstruction from sequence modeling, can scale to 256x256 resolution while outperforming massive, end-to-end architectures.
The HARP Framework
The core philosophy of HARP is simplicity through decoupling. The architecture consists of two distinct stages that allow for specialized optimization of both appearance and motion.
1. High-Fidelity Image Generation (VQ-GAN)
Instead of a standard VQ-VAE, HARP utilizes a VQ-GAN. This model employs perceptual and adversarial losses to compress images into a grid of discrete latent codes (tokens). This approach ensures that even at high compression ratios, the model maintains sharp, realistic details. A significant advantage of this decoupling is that the VQ-GAN can be pre-trained on a general, large-scale dataset (such as ImageNet) and then frozen, providing a robust visual vocabulary for various downstream video tasks.
2. Dynamics Modeling (Causal Transformer)
Once the video frames are converted into sequences of discrete tokens, a causal transformer is trained to predict the next token in the sequence. By operating in this discrete latent space, the transformer effectively learns the "grammar" of environmental changes over time without the burden of pixel-level reconstruction.
Refinements for Long-Term Stability
To bridge the gap between simple latent modeling and high-quality video generation, the authors introduced two key refinements:
- Top-k Sampling: During long-term prediction, autoregressive models can drift into unrealistic latent states. By employing top-k sampling, the model filters out low-probability tokens from the long tail of the distribution, ensuring that the sampled tokens remain within the manifold of realistic images.
- Weak Data Augmentation: To improve the transformer's robustness, the authors used simple translation augmentation (shifting images by a few pixels). This helps the model generalize to motion dynamics without interfering with the frozen VQ-GAN’s ability to reconstruct the scene.
Experimental Results
Benchmarking Performance
HARP was evaluated against several state-of-the-art (SOTA) models across diverse datasets:
- KITTI (Driving): On this real-world driving dataset (128x128), HARP achieved a Fréchet Video Distance (FVD) of 482.9, nearly doubling the performance of the previous SOTA, FitVid (884.5). It also secured the best LPIPS score (0.191), indicating superior perceptual similarity to ground truth.
- BAIR Robot Pushing: On this standard robotics benchmark (64x64), HARP achieved an FVD of 99.3, performing comparably to the Video Transformer (94.0) despite using roughly four times fewer parameters (89M vs. 300M-600M).
- High-Resolution Scaling: HARP successfully demonstrated 256x256 video prediction on the RoboNet and Kinetics-600 datasets. Even at these higher resolutions, the model maintained sharp edges and textures that pixel-space models typically struggle to preserve.
Efficiency and Transferability
One of the most striking findings is the power of pre-training. Using a VQ-GAN pre-trained on ImageNet allowed HARP to predict high-quality frames for the Kinetics-600 dataset (human actions) without needing to train the generator from scratch on video data. This drastically reduces the computational cost of adapting the model to new domains.
Limitations and Future Work
While HARP represents a significant step forward, the authors noted two primary failure modes:
- Interaction Blindness: In some robotics samples, the model may predict a robot arm moving but fail to simulate the resulting physical interaction with an object, leaving the object static.
- Degenerate Predictions: On highly complex datasets like Kinetics-600, the model occasionally falls into a "lazy" state, repeating the initial frame rather than predicting motion—a known challenge in autoregressive modeling under high uncertainty.
Future research may explore larger transformer architectures or deeper integration of action-conditioning to better capture complex physical interactions.
Conclusion
HARP demonstrates that scaling video prediction to high resolutions does not require increasingly complex end-to-end architectures. By combining the high-fidelity reconstruction of VQ-GAN with the sequence-modeling power of causal transformers, HARP achieves state-of-the-art results with greater parameter efficiency and superior visual clarity.
Summary updated