Deep Probabilistic Video Compression

Jun Han, Salvator Lombardo, Christopher Schroers, Stephan Mandt

Abstract

While deep generative models have revolutionized image compression, neural video compression remains a developing field. This paper proposes an end-to-end, deep generative approach to compress temporal sequences. Building upon sequential Variational Autoencoder (VAE) frameworks, the model jointly learns to transform original sequences into lower-dimensional representations, discretize them, and entropy code them based on sequential VAE predictions. Evaluations on public datasets show that while the model is competitive on generic video content, it achieves extreme compression performance when trained on specialized content.

Introduction

Video content currently accounts for approximately 80% of global internet traffic, making efficient compression a critical technological priority. For decades, the industry has relied on "block-based hybrid codecs" such as H.264, H.265, and VP9. While highly optimized, these systems are manually engineered with complex algorithms for motion estimation and residual coding. This research proposes a shift toward fully learned, unsupervised generative models that treat video as a stochastic sequence rather than a collection of independent blocks.

Methodology

The proposed codec utilizes a Sequential Variational Autoencoder (VAE) framework with several key innovations:

  • Latent Variable Separation: The model distinguishes between global variables ($f$), which capture static information like backgrounds, and local variables ($z_t$), which represent frame-by-frame temporal changes.
  • Temporal Prior (LSTM): A Long Short-Term Memory (LSTM) network predicts the probability distribution of the next frame's latent variables based on previous frames, replacing traditional motion vectors.
  • Differentiable Quantization: To enable backpropagation, the model uses uniform noise during training to simulate the quantization (rounding) required for compression.
  • Arithmetic Coding: The learned probability distributions are used by an arithmetic coder to generate a binary bitstream. Higher predictive certainty results in lower bitrates.

Experimental Setup

The model was evaluated on three datasets of increasing complexity at 64x64 resolution:

  1. Sprites: Synthetic animations of game characters, representing low-dimensional specialized content.
  2. BAIR Robot Pushing: Real-world footage of a robot arm with consistent backgrounds.
  3. Kinetics600: A diverse dataset of YouTube videos representing generic, high-complexity content.

Key Results and Insights

Extreme Compression on Specialized Content

The model demonstrates its greatest strength on specialized content. On the Sprites dataset, it achieved significantly higher quality (measured in PSNR) than H.265 and VP9 while requiring nearly 10x fewer bits per pixel. This suggests that generative models can learn the specific "grammar" of a content type to achieve unprecedented efficiency.

Comparison of H.265, VP9, and the proposed model on the Sprites dataset.

Performance and Visual Quality

On the diverse Kinetics dataset, the model remained competitive with traditional codecs at low resolutions. Qualitatively, the generative approach avoids the "blocky" artifacts typical of traditional codecs at low bitrates. Instead, the reconstructions tend to be slightly blurry, a characteristic of VAE-based models that many viewers find less distracting than geometric artifacts.

Entropy Decay

An interesting observation was "entropy decay": the bit cost of encoding local variables decreased over the course of a video segment. This indicates the LSTM successfully "warms up," becoming more accurate in its predictions as it processes more temporal context.

Limitations and Future Work

Currently, the model is limited to 64x64 resolution due to the memory intensity of fully-connected layers used for inference. Future research will focus on fully convolutional architectures to scale to high-definition resolutions. Additionally, the authors see significant potential in developing specialized codecs for predictable environments, such as teleconferencing, where the model can be pre-trained on specific facial and background structures.

Summary updated

Abstract

The usage of deep generative models for image compression has led to impressive performance gains over classical codecs while neural video compression is still in its infancy. Here, we propose an end-to-end, deep generative modeling approach to compress temporal sequences with a focus on video. Our approach builds upon variational autoencoder (VAE) models for sequential data and combines them with recent work on neural image compression. The approach jointly learns to transform the original sequence into a lower-dimensional representation as well as to discretize and entropy code this representation according to predictions of the sequential VAE. Rate-distortion evaluations on small videos from public data sets with varying complexity and diversity show that our model yields competitive results when trained on generic video content. Extreme compression performance is achieved when training the model on specialized content.
BibTeX
@inproceedings{han2019deep,
  author = {Jun Han and Salvator Lombardo and Christopher Schroers and Stephan Mandt},
  title = {{Deep Probabilistic Video Compression}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=hoBNIO5z5f},
}
Figure