Deep Generative Video Compression

Salvator Lombardo, Jun Han, Christopher Schroers, Stephan Mandt

Video accounts for roughly 80% of all internet traffic. For decades, we’ve relied on standards like H.264 and H.265, which are marvels of engineering built on a "block-based hybrid" structure—essentially predicting motion between blocks of pixels and coding the leftovers (residuals). But as we hit the limits of traditional engineering, researchers are asking: can we learn to compress video from scratch?

In this NeurIPS 2019 paper, researchers from Dartmouth, Disney, and UC Irvine propose a new way forward by framing video compression as a deep generative modeling problem.

TL;DR

The authors present an end-to-end neural video codec based on sequential Variational Autoencoders (VAEs). By jointly learning to transform video into a latent space and predicting the probability of future frames, their model achieves "extreme compression" on specialized content and remains competitive with traditional codecs on generic video.

What Problem Does This Paper Address?

Traditional video codecs (like H.264/AVC or VP9) are incredibly complex, hand-engineered systems. They rely on motion estimation to find similarities between frames and then compress the "residual" error. While effective, these methods struggle with fine-grained details at very low bitrates, often resulting in "blocky" artifacts.

Neural image compression has already started beating JPEG and WebP, but video is a harder beast. The challenge is temporal redundancy: how do you efficiently represent the fact that most of a video doesn't change from one millisecond to the next? Most existing neural video attempts still mimic the old "motion + residual" pipeline. This paper seeks to move beyond that by using a purely probabilistic, generative approach.

Methods

The core of the approach is a Sequential Variational Autoencoder (VAE). Here is the breakdown of the "magic" under the hood:

  1. The Latent Split: The model decomposes a video segment into two parts:
  2. Global Latent ($f$): A single variable that captures static information across the whole segment (like a "keyframe" or background).
  3. Local Latents ($z_t$): Per-frame variables that capture the dynamic, changing parts of the scene.
  4. The Predictive Prior: Instead of just sending residuals, the model uses an LSTM-based prior. This LSTM looks at all previous frames to predict the probability distribution of the next latent state.
  5. Entropy Coding: This is the "compression" part. Because the model has a powerful predictive prior, it can assign high probabilities to likely next states. Using an Arithmetic Coder, these high-probability states are stored using very few bits.
  6. Differentiable Quantization: To make the model trainable via backpropagation, the authors use a clever trick: they inject uniform noise during training to simulate the "rounding" (quantization) that happens during actual compression.

Datasets

The authors evaluated their model on three datasets of increasing complexity:

  • Sprites: Simple, low-dimensional animations of video game characters.
  • BAIR: A dataset of a robot arm pushing objects on a table. It’s more realistic but has a consistent background and limited motion types.
  • Kinetics600: A highly diverse dataset of YouTube videos. The authors downsampled these to 64x64 resolution to fit their experimental constraints.

Key Insights and Ideas

  • Generative > Residuals: Unlike traditional codecs that code the difference between frames, this model learns the probability of the next frame. This allows it to "imagine" details that traditional codecs would simply blur out or blockify at low bitrates.
  • The Power of Specialization: The most striking "aha" moment is how well the model performs when trained on specific content. If the model "knows" what a Sprite character looks like, it can compress that video to a tiny fraction of the size required by H.265 while maintaining much higher visual quality.
  • Global States Matter: By separating the "global" (static) and "local" (dynamic) information, the model naturally learns to avoid re-sending background information, mimicking the efficiency of keyframes in a more flexible, learned way.

Key Results

  • Extreme Compression: On the Sprites dataset, the model achieved a PSNR (quality metric) of 44.6 dB at only 0.06 bits per pixel (bpp). For comparison, VP9 required nearly 10x the bitrate (0.57 bpp) to reach a much lower quality of 26.0 dB.
Visual comparison on the Sprites dataset showing the proposed model outperforming H.265 and VP9 at a fraction of the bitrate.
  • Competitive on Generic Content: On the diverse Kinetics dataset, the neural codec was competitive with H.264 and H.265, proving that the generative approach isn't just for "toy" datasets.
  • Visual Fidelity: Qualitatively, the model avoids the "blocky" artifacts of H.264. While it can become slightly "blurry" at extremely low bitrates, it preserves the geometric structure of objects much better than traditional methods.

Limitations and Future Work

  • Resolution Constraints: The current experiments were limited to 64x64 resolution. Scaling this to 1080p or 4K is a massive challenge because the fully-connected layers used for global/local inference are memory-intensive.
  • Computational Complexity: Neural decoding is currently much slower than the dedicated hardware decoders used for H.264.
  • Future Directions: The authors suggest moving toward fully convolutional architectures to handle larger resolutions and exploring even more powerful predictive models to further drive down the bitrate.

Reference: Han, J., Lombardo, S., Schroers, C., & Mandt, S. (2019). Deep Generative Video Compression. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019).

Summary updated

Abstract

The usage of deep generative models for image compression has led to impressive performance gains over classical codecs while neural video compression is still in its infancy. Here, we propose an end-to-end, deep generative modeling approach to compress temporal sequences with a focus on video. Our approach builds upon variational autoencoder (VAE) models for sequential data and combines them with recent work on neural image compression. The approach jointly learns to transform the original sequence into a lower-dimensional representation as well as to discretize and entropy code this representation according to predictions of the sequential VAE. Rate-distortion evaluations on small videos from public data sets with varying complexity and diversity show that our model yields competitive results when trained on generic video content. Extreme compression performance is achieved when training the model on specialized content.
BibTeX
@inproceedings{lombardo2019deep,
  author = {Salvator Lombardo and Jun Han and Christopher Schroers and Stephan Mandt},
  title = {{Deep Generative Video Compression}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=hoBNIO5z5f},
}
Figure