Content-Adaptive Motion Rate Adaption for Learned Video Compression

Chih-Hsuan Lin, Yi-Hsin Chen, Wen-Hsiao Peng

Abstract

This paper introduces an online motion rate adaptation scheme for learned video compression, with the aim of achieving content-adaptive coding on individual test sequences to mitigate the domain gap between training and test data. It features a patch-level bit allocation map, termed the $\alpha$-map, to trade off between the bit rates for motion and inter-frame coding in a spatially-adaptive manner. We optimize the $\alpha$-map through an online back-propagation scheme at inference time. Moreover, we incorporate a look-ahead mechanism to consider its impact on future frames. Extensive experimental results confirm that the proposed scheme, when integrated into a conditional learned video codec, is able to adapt motion bit rate effectively, showing much improved rate-distortion performance particularly on test sequences with complicated motion characteristics.

Introduction

Standard learned video compression models are typically trained on large datasets and then fixed. However, a "domain gap" often exists between the training data and the specific video a user wants to compress. A model optimized for generic motion might struggle with a sequence featuring unique textures or highly complex movement, such as zooming or rotation.

Most existing codecs use a fixed strategy for bit allocation between motion coding (describing how things move) and inter-frame/residual coding (describing the differences between frames). This paper addresses the lack of spatial and temporal flexibility in this allocation, which often leads to sub-optimal performance on individual test sequences. The authors propose a content-adaptive "$\alpha$-map" that allows learned video codecs to dynamically reallocate bits between these two components at inference time.

Methodology

The core of the proposed method is built upon a Conditional Augmented Normalizing Flow (CANF) framework. The system introduces several key innovations to enable content-adaptive compression:

  • The $\alpha$-map: This is a patch-level bit allocation map (typically sized at 1/64th of the frame resolution). Each value $\alpha_i$ in the map acts as a "knob" that trades off the bit rate between motion coding and inter-frame coding for a specific $64 \times 64$ pixel patch.
  • Spatial Feature Transform (SFT): To make the codec aware of the $\alpha$-map, SFT layers are incorporated. These layers apply spatially-adaptive affine transformations to the latent features of the network, conditioned on the $\alpha$-map values.
  • Online Back-propagation: Instead of a purely static model, the system optimizes the $\alpha$-map at inference time. It performs gradient descent steps to find the $\alpha$ values that minimize the rate-distortion cost for the specific frame being processed.
  • Look-ahead Mechanism: To avoid "greedy" optimization that might negatively impact future frames, a sliding window approach is used. The $\alpha$-map for the current frame is optimized by considering its impact on both the current and subsequent frames.
Overview of the proposed video compression framework, highlighting the integration of the alpha-map within the Motion and Inter-frame Coders.

Experimental Setup

The model was trained using the Vimeo-90k dataset, a standard benchmark containing over 90,000 video sequences. Evaluation was conducted on several diverse datasets to test robustness:

  • UVG (Ultra Video Group): High-quality 4K sequences.
  • HEVC Class D: Standard sequences used in traditional video coding benchmarks.
  • CLIC’22: Challenging sequences featuring complex motions like fast movement, zooming, and rotation.

Key Insights and Results

The primary insight of this work is that different parts of a frame require different bit priorities. A static area needs almost no motion bits, while a complex moving object might benefit from spending more bits on motion to make the inter-frame prediction more accurate, thereby reducing the bits needed for the residual.

Visualization of alpha-maps, bit allocation, and compressed optical flow, demonstrating the spatial adaptivity of the proposed scheme.

Performance Gains

  • BD-Rate Savings: The proposed method achieved 2% to 5% BD-rate savings compared to a strong anchor model without the $\alpha$-map.
  • Complex Motion Handling: The method generally outperformed DCVC, a state-of-the-art learned video codec, particularly on sequences with complex motion characteristics.
  • Qualitative Improvement: Visualizations show that the optimized $\alpha$-map allows the compressed optical flow to preserve much finer details in complex regions compared to the baseline, leading to better overall reconstruction.
  • Temporal Consistency: Including the look-ahead mechanism consistently provided better results than optimizing frame-by-frame, proving that bit allocation decisions have significant temporal dependencies.

Conclusion and Future Work

While the proposed scheme offers significant rate-distortion improvements by bridging the gap between general training and specific testing, it introduces computational overhead. The primary drawback is the use of online back-propagation at inference time, which increases encoding latency compared to standard feed-forward models. Additionally, memory constraints necessitated downscaling high-resolution sequences for optimization.

Future research may focus on more sophisticated look-ahead strategies or methods to accelerate the online optimization process, making content-adaptive learned video compression more practical for real-time applications.

Summary updated

Abstract

This paper introduces an online motion rate adaptation scheme for learned video compression, with the aim of achieving content-adaptive coding on individual test sequences to mitigate the domain gap between training and test data. It features a patch-level bit allocation map, termed the $\alpha$-map, to trade off between the bit rates for motion and inter-frame coding in a spatially-adaptive manner. We optimize the $\alpha$-map through an online back-propagation scheme at inference time. Moreover, we incorporate a look-ahead mechanism to consider its impact on future frames. Extensive experimental results confirm that the proposed scheme, when integrated into a conditional learned video codec, is able to adapt motion bit rate effectively, showing much improved rate-distortion performance particularly on test sequences with complicated motion characteristics.
BibTeX
@inproceedings{lin2022content,
  author = {Chih-Hsuan Lin and Yi-Hsin Chen and Wen-Hsiao Peng},
  title = {{Content-Adaptive Motion Rate Adaption for Learned Video Compression}},
  booktitle = {PCS},
  year = {2022},
  url = {https://openreview.net/forum?id=gdx71ZijnF},
}
Figure