Omra: Online Motion Resolution Adaptation To Remedy Domain Shift in Learned Hierarchical B-Frame Coding

Zong-Lin Gao, Sang NguyenQuang, Wen-Hsiao Peng, Xiem HoangVanICIP 2024 · DOI

TL;DR

This paper introduces OMRA, a novel method to improve learned hierarchical B-frame video compression. OMRA tackles the "domain shift" problem, where models trained on short video clips struggle with longer ones, by adaptively downsampling frames during inference. This simple yet effective technique significantly boosts compression efficiency without retraining existing models.

What Problem Does This Paper Address?

Learned video compression, particularly for B-frames (which use both past and future frames for prediction), promises higher efficiency than traditional methods. However, a major hurdle is the "domain shift" problem. Current learned B-frame codecs are typically trained on short video sequences with small Groups of Pictures (GOPs), often just 5-7 frames. When these trained models are used to compress longer sequences with much larger GOPs (e.g., 32 frames), the motion estimation networks, optimized for short-range predictions, fail to handle the larger motions present in long GOPs. This leads to poor prediction quality and reduced compression performance. Creating new, large-scale datasets for training on long GOPs is computationally expensive and time-consuming, making this a significant challenge for practical deployment.

Methods

The core of OMRA is an online motion resolution adaptation strategy. Instead of retraining the entire B-frame codec, OMRA operates during the inference (encoding) phase. For each B-frame, it performs an exhaustive search for the optimal downsampling factor (S ∈ {1, 2, 4, 8}) to apply to the current frame and its reference frames before motion estimation. The goal is to reduce large motions in high-resolution frames to smaller motions in their low-resolution counterparts, making them manageable for the pre-trained motion estimation network.

The process involves:

  1. Downsampling: The target and reference frames are downsampled by a chosen factor S.
  2. Motion Estimation: A low-resolution motion estimation network (MENet) estimates optical flow maps between the downsampled frames.
  3. Motion Coding: These low-resolution flow maps are compressed and sent to the decoder.
  4. Upsampling & Compensation: The decoded low-resolution flow maps are upsampled to the original resolution. These are then used with the high-resolution reference frames to generate a temporal predictor for the target frame.
  5. Rate-Distortion Optimization: The key is that for each frame, OMRA calculates the rate-distortion cost for each downsampling factor and selects the one that yields the best trade-off between distortion (reconstruction quality) and bitrate. This ensures the adaptation is tailored to the specific content of each frame.

Datasets

The paper evaluates OMRA on standard video datasets commonly used for learned video compression research:

  • UVG dataset: A dataset of high-quality video sequences.
  • HEVC Class B dataset: A standard set of sequences used for evaluating HEVC (High Efficiency Video Coding).
  • MCL-JCV dataset: Another dataset for video quality assessment.

The models used for evaluation (B-CANF and MaskCRT) were trained on the Vimeo-90k dataset, which typically uses small GOPs (5 frames). The evaluation itself was performed with a larger GOP size of 32 frames to simulate the domain shift scenario.

Key Insights and Ideas

  • Domain Shift is a Major Bottleneck: The paper strongly emphasizes that the mismatch between training (small GOPs) and testing (large GOPs) is a primary reason why learned B-frame codecs underperform their P-frame counterparts, despite theoretical advantages.
  • Resolution Adaptation is Key: The core insight is that by adaptively downsampling frames, large motions can be effectively "shrunk" to fit the capabilities of a motion estimation network trained on smaller motions. This is a clever way to bridge the gap without retraining.
  • Per-Frame Optimization is Crucial: OMRA's strength lies in its per-frame rate-distortion optimization. This allows it to dynamically choose the best downsampling factor based on the local content and motion characteristics, rather than applying a fixed strategy.
  • Flow Domain Adaptation is Superior: The paper's ablation studies suggest that adapting the resolution in the flow domain (downsampling frames, estimating low-res flow, then upsampling flow) is more effective than adapting in the pixel domain or upsampling after warping.

Key Results

  • Significant BD-rate Improvements: OMRA consistently improves the compression performance of two state-of-the-art learned B-frame codecs (B-CANF and MaskCRT).
  • MaskCRT achieved BD-rate improvements of 10.6% on HEVC Class B, 19.2% on UVG, and 17.0% on MCL-JCV.
  • B-CANF saw improvements of 6.5% on HEVC Class B, 12.6% on UVG, and 8.1% on MCL-JCV.
  • Fast-Motion Sequences Benefit Most: The gains are particularly pronounced on fast-motion sequences (e.g., BasketballDrive, Jockey), where the domain shift is most severe.
  • Subjective Quality Improvement: Visualizations show that OMRA produces better-quality warped frames and temporal predictors, leading to improved overall reconstruction quality.
  • Negligible Impact on Decoding Complexity: While encoding complexity increases due to the search, the decoding complexity is minimally affected.

Limitations and Future Work

  • Encoding Complexity: The primary limitation acknowledged is the increased encoding time due to the exhaustive search for the optimal downsampling factor for each frame.
  • Future Work: The authors explicitly mention that predicting the optimal downsampling factor on a per-frame basis without an exhaustive search is a key area for future research. This would help mitigate the encoding complexity issue.

Summary updated

BibTeX
@inproceedings{gao2024omra,
  author = {Zong-Lin Gao and Sang NguyenQuang and Wen-Hsiao Peng and Xiem HoangVan},
  title = {{Omra: Online Motion Resolution Adaptation To Remedy Domain Shift in Learned Hierarchical B-Frame Coding}},
  booktitle = {ICIP},
  year = {2024},
  doi = {10.1109/icip51287.2024.10647370},
}
Figure