Abstract
This paper introduces an online motion rate adaptation scheme for learned video compression, with the aim of achieving content-adaptive coding on individual test sequences to mitigate the domain gap between training and test data. It features a patch-level bit allocation map, termed the $\alpha$-map, to trade off between the bit rates for motion and inter-frame coding in a spatially-adaptive manner. We optimize the $\alpha$-map through an online back-propagation scheme at inference time. Moreover, we incorporate a look-ahead mechanism to consider its impact on future frames. Extensive experimental results confirm that the proposed scheme, when integrated into a conditional learned video codec, is able to adapt motion bit rate effectively, showing much improved rate-distortion performance particularly on test sequences with complicated motion characteristics.
Introduction
Standard learned video compression models are typically trained on large datasets and then fixed. However, a "domain gap" often exists between the training data and the specific video a user wants to compress. A model optimized for generic motion might struggle with a sequence featuring unique textures or highly complex movement, such as zooming or rotation.
Most existing codecs use a fixed strategy for bit allocation between motion coding (describing how things move) and inter-frame/residual coding (describing the differences between frames). This paper addresses the lack of spatial and temporal flexibility in this allocation, which often leads to sub-optimal performance on individual test sequences. The authors propose a content-adaptive "$\alpha$-map" that allows learned video codecs to dynamically reallocate bits between these two components at inference time.
Methodology
The core of the proposed method is built upon a Conditional Augmented Normalizing Flow (CANF) framework. The system introduces several key innovations to enable content-adaptive compression:
- The $\alpha$-map: This is a patch-level bit allocation map (typically sized at 1/64th of the frame resolution). Each value $\alpha_i$ in the map acts as a "knob" that trades off the bit rate between motion coding and inter-frame coding for a specific $64 \times 64$ pixel patch.
- Spatial Feature Transform (SFT): To make the codec aware of the $\alpha$-map, SFT layers are incorporated. These layers apply spatially-adaptive affine transformations to the latent features of the network, conditioned on the $\alpha$-map values.
- Online Back-propagation: Instead of a purely static model, the system optimizes the $\alpha$-map at inference time. It performs gradient descent steps to find the $\alpha$ values that minimize the rate-distortion cost for the specific frame being processed.
- Look-ahead Mechanism: To avoid "greedy" optimization that might negatively impact future frames, a sliding window approach is used. The $\alpha$-map for the current frame is optimized by considering its impact on both the current and subsequent frames.
Experimental Setup
The model was trained using the Vimeo-90k dataset, a standard benchmark containing over 90,000 video sequences. Evaluation was conducted on several diverse datasets to test robustness:
- UVG (Ultra Video Group): High-quality 4K sequences.
- HEVC Class D: Standard sequences used in traditional video coding benchmarks.
- CLIC’22: Challenging sequences featuring complex motions like fast movement, zooming, and rotation.
Key Insights and Results
The primary insight of this work is that different parts of a frame require different bit priorities. A static area needs almost no motion bits, while a complex moving object might benefit from spending more bits on motion to make the inter-frame prediction more accurate, thereby reducing the bits needed for the residual.
Performance Gains
- BD-Rate Savings: The proposed method achieved 2% to 5% BD-rate savings compared to a strong anchor model without the $\alpha$-map.
- Complex Motion Handling: The method generally outperformed DCVC, a state-of-the-art learned video codec, particularly on sequences with complex motion characteristics.
- Qualitative Improvement: Visualizations show that the optimized $\alpha$-map allows the compressed optical flow to preserve much finer details in complex regions compared to the baseline, leading to better overall reconstruction.
- Temporal Consistency: Including the look-ahead mechanism consistently provided better results than optimizing frame-by-frame, proving that bit allocation decisions have significant temporal dependencies.
Conclusion and Future Work
While the proposed scheme offers significant rate-distortion improvements by bridging the gap between general training and specific testing, it introduces computational overhead. The primary drawback is the use of online back-propagation at inference time, which increases encoding latency compared to standard feed-forward models. Additionally, memory constraints necessitated downscaling high-resolution sequences for optimization.
Future research may focus on more sophisticated look-ahead strategies or methods to accelerate the online optimization process, making content-adaptive learned video compression more practical for real-time applications.
Summary updated