Content-Adaptive Motion Rate Adaption For Learned Video Compression

Chih Hsuan Lin, Yi-Hsin Chen, Wen-Hsiao Peng

TL;DR

This paper introduces a content-adaptive "$\alpha$-map" that allows learned video codecs to dynamically reallocate bits between motion estimation and inter-frame coding at inference time. By using online back-propagation and a look-ahead mechanism, the system adapts to the specific characteristics of a video sequence, achieving significant rate-distortion improvements—especially on videos with complex motion.

What Problem Does This Paper Address?

Standard learned video compression models are typically trained on large datasets and then fixed. However, a "domain gap" often exists between the training data and the specific video a user wants to compress. A model optimized for generic motion might struggle with a sequence featuring unique textures or highly complex movement (like zooming or rotation).

Most existing codecs use a fixed strategy for how many bits to spend on describing how things move (motion coding) versus what the differences are between frames (inter-frame/residual coding). This paper addresses the lack of spatial and temporal flexibility in this bit allocation, which often leads to sub-optimal performance on individual test sequences.

Methods

The authors build their method on top of a Conditional Augmented Normalizing Flow (CANF) framework. Their core contributions include:

  • The $\alpha$-map: A patch-level bit allocation map (sized at 1/64th of the frame resolution). Each value $\alpha_i$ in the map acts as a "knob" that trades off the bit rate between motion coding and inter-frame coding for a specific $64 \times 64$ pixel patch.
  • Spatial Feature Transform (SFT): To make the codec "aware" of the $\alpha$-map, the authors incorporate SFT layers. these layers apply spatially-adaptive affine transformations to the latent features of the network, conditioned on the values in the $\alpha$-map.
  • Online Back-propagation: Instead of just using the model as-is, the system optimizes the $\alpha$-map at inference time. It performs a few steps of gradient descent to find the $\alpha$ values that minimize the rate-distortion cost for the specific frame being processed.
  • Look-ahead Mechanism: To avoid "greedy" optimization that might hurt future frames, the authors implement a sliding window approach. The $\alpha$-map for the current frame is optimized by considering its impact on both the current frame and the subsequent frame.

Datasets

  • Training: The model was trained using the Vimeo-90k dataset, a standard benchmark containing over 90,000 video sequences.
  • Evaluation: The authors tested their approach on several diverse datasets:
  • UVG (Ultra Video Group): High-quality 4K sequences (downscaled for memory reasons).
  • HEVC Class D: Standard sequences used in traditional video coding standards.
  • CLIC’22: Challenging sequences from the Workshop and Challenge on Learned Image Compression, featuring complex motions like fast movement, zooming, and rotation.

Key Insights and Ideas

  • Spatially-Adaptive Bit Allocation: The "aha moment" is realizing that different parts of a frame need different bit priorities. A static area needs almost no motion bits, while a complex moving object might benefit from spending more bits on motion to make the inter-frame prediction much easier.
  • Inference-Time Optimization: By treating the $\alpha$-map as a set of learnable parameters during compression (rather than just weights in a fixed network), the codec can "overfit" slightly to the specific video it is seeing, bridging the gap between general training and specific testing.
  • The Motion-Residual Trade-off: The paper demonstrates that increasing the motion bit rate can actually decrease the total bit rate if it leads to a significantly more accurate temporal prediction, reducing the bits needed for the residual.

Key Results

  • BD-Rate Savings: The proposed method achieved 2% to 5% BD-rate savings (meaning it provides the same quality at a lower bit rate) compared to a strong anchor model without the $\alpha$-map.
  • Superiority over DCVC: The method generally outperformed DCVC, a state-of-the-art learned video codec, particularly on sequences with complex motion.
  • Qualitative Improvement: Visualizations show that the optimized $\alpha$-map allows the compressed optical flow to preserve much finer details in complex regions compared to the baseline, leading to better overall reconstruction.
  • Look-ahead Gains: Including the look-ahead mechanism consistently provided better results than optimizing frame-by-frame, proving that bit allocation decisions have temporal dependencies.

Limitations and Future Work

  • Computational Complexity: The primary drawback is the use of online back-propagation at inference time. Performing gradient descent during the encoding process is computationally expensive and significantly increases encoding time compared to standard feed-forward models.
  • Memory Constraints: The authors noted they had to downscale high-resolution sequences (like UVG and CLIC-MIX) to fit the $\alpha$-map optimization within limited GPU memory.
  • Future Directions: The authors suggest that future work could involve more sophisticated look-ahead strategies or finding ways to speed up the online optimization process to make it more practical for real-time applications.

Not discussed in the paper: The specific increase in encoding time (latency) caused by the back-propagation steps.

Summary updated

Abstract

This paper introduces an online motion rate adaptation scheme for learned video compression, with the aim of achieving content-adaptive coding on individual test sequences to mitigate the domain gap between training and test data. It features a patch-level bit allocation map, termed the $\alpha-$map, to trade off between the bit rates for motion and inter-frame coding in a spatially-adaptive manner. We optimize the $\alpha-$map through an online back-propagation scheme at inference time. Moreover, we incorporate a look-ahead mechanism to consider its impact on future frames. Extensive experimental results confirm that the proposed scheme, when integrated into a conditional learned video codec, is able to adapt motion bit rate effectively, showing much improved rate-distortion performance particularly on test sequences with complicated motion characteristics.
BibTeX
@inproceedings{lin2022content,
  author = {Chih Hsuan Lin and Yi-Hsin Chen and Wen-Hsiao Peng},
  title = {{Content-Adaptive Motion Rate Adaption For Learned Video Compression}},
  booktitle = {PCS},
  year = {2022},
  url = {https://openreview.net/forum?id=gdx71ZijnF},
}

Loading related content...

Figure