Abstract
While deep generative models have revolutionized image compression, neural video compression remains a developing field. This paper proposes an end-to-end, deep generative approach to compress temporal sequences. Building upon sequential Variational Autoencoder (VAE) frameworks, the model jointly learns to transform original sequences into lower-dimensional representations, discretize them, and entropy code them based on sequential VAE predictions. Evaluations on public datasets show that while the model is competitive on generic video content, it achieves extreme compression performance when trained on specialized content.
Introduction
Video content currently accounts for approximately 80% of global internet traffic, making efficient compression a critical technological priority. For decades, the industry has relied on "block-based hybrid codecs" such as H.264, H.265, and VP9. While highly optimized, these systems are manually engineered with complex algorithms for motion estimation and residual coding. This research proposes a shift toward fully learned, unsupervised generative models that treat video as a stochastic sequence rather than a collection of independent blocks.
Methodology
The proposed codec utilizes a Sequential Variational Autoencoder (VAE) framework with several key innovations:
- Latent Variable Separation: The model distinguishes between global variables ($f$), which capture static information like backgrounds, and local variables ($z_t$), which represent frame-by-frame temporal changes.
- Temporal Prior (LSTM): A Long Short-Term Memory (LSTM) network predicts the probability distribution of the next frame's latent variables based on previous frames, replacing traditional motion vectors.
- Differentiable Quantization: To enable backpropagation, the model uses uniform noise during training to simulate the quantization (rounding) required for compression.
- Arithmetic Coding: The learned probability distributions are used by an arithmetic coder to generate a binary bitstream. Higher predictive certainty results in lower bitrates.
Experimental Setup
The model was evaluated on three datasets of increasing complexity at 64x64 resolution:
- Sprites: Synthetic animations of game characters, representing low-dimensional specialized content.
- BAIR Robot Pushing: Real-world footage of a robot arm with consistent backgrounds.
- Kinetics600: A diverse dataset of YouTube videos representing generic, high-complexity content.
Key Results and Insights
Extreme Compression on Specialized Content
The model demonstrates its greatest strength on specialized content. On the Sprites dataset, it achieved significantly higher quality (measured in PSNR) than H.265 and VP9 while requiring nearly 10x fewer bits per pixel. This suggests that generative models can learn the specific "grammar" of a content type to achieve unprecedented efficiency.
Performance and Visual Quality
On the diverse Kinetics dataset, the model remained competitive with traditional codecs at low resolutions. Qualitatively, the generative approach avoids the "blocky" artifacts typical of traditional codecs at low bitrates. Instead, the reconstructions tend to be slightly blurry, a characteristic of VAE-based models that many viewers find less distracting than geometric artifacts.
Entropy Decay
An interesting observation was "entropy decay": the bit cost of encoding local variables decreased over the course of a video segment. This indicates the LSTM successfully "warms up," becoming more accurate in its predictions as it processes more temporal context.
Limitations and Future Work
Currently, the model is limited to 64x64 resolution due to the memory intensity of fully-connected layers used for inference. Future research will focus on fully convolutional architectures to scale to high-definition resolutions. Additionally, the authors see significant potential in developing specialized codecs for predictable environments, such as teleconferencing, where the model can be pre-trained on specific facial and background structures.
Summary updated