TL;DR
This paper introduces a novel end-to-end learned video compression framework that adapts to different video content and operates at multiple bitrates using a single model. By employing multi-scale deformable alignment and content-adaptive inference, it achieves state-of-the-art rate-distortion performance, outperforming previous learned methods and even rivaling traditional codecs like VVC.
What Problem Does This Paper Address?
Traditional video compression codecs (like H.265/HEVC and H.266/VVC) are highly sophisticated and can adapt their compression strategies to the specific content of a video. This adaptability is crucial for achieving good compression efficiency. In contrast, current end-to-end learned video compression models, while optimized for rate-distortion performance, often lack this ability to adapt to diverse video content. This limitation prevents them from reaching the performance levels of classical codecs. The paper aims to bridge this gap by developing a learned compression model that can dynamically adjust its inference process based on the video content, while also supporting flexible bitrate control.
Methods
The core of the proposed method lies in two main innovations:
- Multi-scale Deformable Alignment and Conditional Coding: Instead of traditional pixel-level motion estimation and compensation, this work operates in the feature space. It uses a multi-scale deformable alignment scheme to match features from reference frames to the current frame. This is combined with multi-scale conditional coding, where the model learns to predict and compress the current frame's features conditioned on the aligned features from past and future frames. This approach allows for more robust handling of complex motion and details.
- Content-Adaptive Inference: To address the lack of content adaptability, the authors propose an "online encoder update strategy." During inference, specific parts of the encoder are fine-tuned for each frame to adapt to the current video's characteristics (e.g., motion patterns, resolution). This allows the model to dynamically adjust its behavior without requiring retraining for every new video type.
- Flexible-Rate Control: A "gain unit" is employed, which enables a single trained model to operate across a range of rate-distortion points. This is achieved by scaling latent representations before quantization, effectively controlling the quantization bin sizes and thus the bitrate. This gain unit is also used to balance bit allocation between intra-coded and bi-directionally coded frames.
Datasets
- Training: Vimeo90k septuplet dataset.
- Testing: 1080p UVG dataset.
Key Insights and Ideas
- Feature-space deformable alignment is powerful: Moving motion compensation from the pixel domain to the feature domain, coupled with deformable convolutions, allows for more sophisticated and robust alignment, especially in the presence of occlusions or complex motion.
- Content adaptation is key for learned codecs: The experimental results strongly suggest that enabling learned codecs to adapt their inference process online significantly boosts performance, bringing them closer to or even surpassing traditional codecs.
- A single model for multiple bitrates is achievable: The gain unit mechanism effectively demonstrates that a single learned model can be trained to achieve state-of-the-art performance across a range of bitrates, simplifying deployment and training.
- Bi-directional compression benefits from learned techniques: Extending learned techniques like deformable alignment and conditional coding to bi-directional (B-frame) compression unlocks significant performance gains, particularly in challenging sequences.
Key Results
The paper reports state-of-the-art rate-distortion performance, outperforming all prior learned video compression methods. Specifically:
- The proposed model achieves superior PSNR vs. bits-per-pixel (bpp) performance compared to existing learned codecs.
- When compared to the VTM 18.0 (a state-of-the-art traditional codec), the proposed method with content-adaptive inference shows significant average BD-BR (Bit-rate Distortion - Bitrate) improvements, particularly on sequences with complex motion or occlusions. For instance, on the Honeybee sequence, it significantly exceeds the performance of a leading learned sequential coding model [14].
- The content-adaptive inference strategy provides noticeable performance gains, especially for sequences with complex motion patterns not well-represented in the training data.
Limitations and Future Work
- Limitations: While the method shows strong performance, the paper notes that the benefits of content adaptation can vary depending on the video's motion complexity and how well it's represented in the training data. For simpler motion patterns, the gains are smaller.
- Future Work: The authors plan to further investigate better modeling of complex motions as a direction for future research.
Summary updated