Introduction
In video compression, the entropy model predicts the probability distribution of data to minimize the bitstream size. While neural video codecs have shown promise, they often struggle with three key issues: failing to fully exploit temporal redundancy, relying on slow serial spatial models that hinder GPU parallelism, and requiring separate models for different bitrates. This paper introduces a hybrid spatial-temporal entropy model that addresses these challenges, becoming the first end-to-end neural video codec to outperform the H.266 (VTM) standard.
Methods
The proposed architecture integrates three core innovations to enhance efficiency and versatility.
- Latent Prior (Temporal): The model leverages the quantized latent representation of the previous frame as a prior for the current one. This "propagation chain" allows the network to implicitly learn long-range temporal dependencies, moving beyond simple motion vector compensation.
- Dual Spatial Prior (Spatial): To maintain high parallel processing speeds on GPUs, the authors developed a dual spatial prior. By splitting channels and processing even and odd positions simultaneously, the model doubles the available spatial context without the latency of traditional auto-regressive models.
- Multi-Granularity Quantization: This mechanism enables single-model rate adjustment through three levels: global (user-defined), channel-wise (learned feature importance), and spatial-channel-wise (content-adaptive maps).
Key Insights
Parallelism without Compromise
The researchers demonstrated that the speed of checkerboard models and the accuracy of serial models are not mutually exclusive. Their channel-splitting approach provides rich context while maintaining high GPU utilization.
Latents as Memory
Using compressed latent states as a prior is more effective than using raw pixels. It allows the network to propagate high-level semantic information through the compression chain, improving prediction accuracy for subsequent frames.
Content-Adaptive Bit Allocation
The entropy model automatically learns to prioritize "important" visual information. By generating its own quantization steps, the codec allocates more bits to complex moving objects and fewer to static backgrounds.
Results
The model was evaluated on standard benchmarks including UVG, MCL-JCV, and HEVC Common Test Conditions.
- Performance: On the UVG dataset, the model achieved an 18.2% bitrate saving over H.266 (VTM 13.2) in PSNR, and a 35.1% saving in MS-SSIM.
- State-of-the-Art: It significantly outperformed previous neural codecs like DCVC and Sheng 2021.
- Efficiency: The parallel-friendly design allows for processing 1080p frames in 500–900ms on a V100 GPU, a significant improvement over previous high-accuracy serial models.
Limitations and Future Work
Despite its performance, the "W-Net" structure in the frame generator remains computationally intensive. Current decoding speeds (approx. 2 FPS for 1080p) are not yet suitable for real-time applications. Future research will focus on optimizing the generator architecture and exploring deeper propagation chains to further enhance compression gains.
Summary updated