Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression

Jiahao Li, Bin Li, Yan Lu

Introduction

In video compression, the entropy model predicts the probability distribution of data to minimize the bitstream size. While neural video codecs have shown promise, they often struggle with three key issues: failing to fully exploit temporal redundancy, relying on slow serial spatial models that hinder GPU parallelism, and requiring separate models for different bitrates. This paper introduces a hybrid spatial-temporal entropy model that addresses these challenges, becoming the first end-to-end neural video codec to outperform the H.266 (VTM) standard.

Methods

The proposed architecture integrates three core innovations to enhance efficiency and versatility.

Overview of the proposed neural video compression framework featuring temporal context mining and multi-granularity quantization.
  1. Latent Prior (Temporal): The model leverages the quantized latent representation of the previous frame as a prior for the current one. This "propagation chain" allows the network to implicitly learn long-range temporal dependencies, moving beyond simple motion vector compensation.
  2. Dual Spatial Prior (Spatial): To maintain high parallel processing speeds on GPUs, the authors developed a dual spatial prior. By splitting channels and processing even and odd positions simultaneously, the model doubles the available spatial context without the latency of traditional auto-regressive models.
  3. Multi-Granularity Quantization: This mechanism enables single-model rate adjustment through three levels: global (user-defined), channel-wise (learned feature importance), and spatial-channel-wise (content-adaptive maps).

Key Insights

Parallelism without Compromise

The researchers demonstrated that the speed of checkerboard models and the accuracy of serial models are not mutually exclusive. Their channel-splitting approach provides rich context while maintaining high GPU utilization.

Latents as Memory

Using compressed latent states as a prior is more effective than using raw pixels. It allows the network to propagate high-level semantic information through the compression chain, improving prediction accuracy for subsequent frames.

Content-Adaptive Bit Allocation

The entropy model automatically learns to prioritize "important" visual information. By generating its own quantization steps, the codec allocates more bits to complex moving objects and fewer to static backgrounds.

Comparison of bit allocation with and without spatial-channel-wise quantization ($qs_t^{sc}$), showing a reduction from 0.065 to 0.056 BPP with improved quality.

Results

The model was evaluated on standard benchmarks including UVG, MCL-JCV, and HEVC Common Test Conditions.

  • Performance: On the UVG dataset, the model achieved an 18.2% bitrate saving over H.266 (VTM 13.2) in PSNR, and a 35.1% saving in MS-SSIM.
  • State-of-the-Art: It significantly outperformed previous neural codecs like DCVC and Sheng 2021.
  • Efficiency: The parallel-friendly design allows for processing 1080p frames in 500–900ms on a V100 GPU, a significant improvement over previous high-accuracy serial models.

Limitations and Future Work

Despite its performance, the "W-Net" structure in the frame generator remains computationally intensive. Current decoding speeds (approx. 2 FPS for 1080p) are not yet suitable for real-time applications. Future research will focus on optimizing the generator architecture and exploring deeper propagation chains to further enhance compression gains.

Summary updated

Abstract

For neural video codec, it is critical, yet challenging, to design an efficient entropy model which can accurately predict the probability distribution of the quantized latent representation. However, most existing video codecs directly use the ready-made entropy model from image codec to encode the residual or motion, and do not fully leverage the spatial-temporal characteristics in video. To this end, this paper proposes a powerful entropy model which efficiently captures both spatial and temporal dependencies. In particular, we introduce the latent prior which exploits the correlation among the latent representation to squeeze the temporal redundancy. Meanwhile, the dual spatial prior is proposed to reduce the spatial redundancy in a parallel-friendly manner. In addition, our entropy model is also versatile. Besides estimating the probability distribution, our entropy model also generates the quantization step at spatial-channel-wise. This content-adaptive quantization mechanism not only helps our codec achieve the smooth rate adjustment in single model but also improves the final rate-distortion performance by dynamic bit allocation. Experimental results show that, powered by the proposed entropy model, our neural codec can achieve 18.2% bitrate saving on UVG dataset when compared with H.266 (VTM) using the highest compression ratio configuration. It makes a new milestone in the development of neural video codec. The codes are at https://github.com/microsoft/DCVC.
BibTeX
@inproceedings{li2022hybrid,
  author = {Jiahao Li and Bin Li and Yan Lu},
  title = {{Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3547845},
}
Figure