End-to-End Rate-Distortion Optimization for Bi-Directional Learned Video Compression

Mustafa Akin Yilmaz, Ahmet Murat Tekalp

TL;DR

This paper introduces a novel end-to-end learned video compression framework that leverages hierarchical, bi-directional motion compensation, optimizing the entire pipeline for rate-distortion performance across groups of pictures (GOPs). This approach significantly outperforms previous learned sequential video compression methods and approaches the performance of traditional codecs like x265.

What Problem Does This Paper Address?

Traditional video compression codecs (like H.264/H.265) rely on hand-engineered modules for motion estimation, transform, quantization, and entropy coding. While effective, these components are optimized individually due to the combinatorial complexity of optimizing the entire system end-to-end. This limitation prevents achieving the theoretical best rate-distortion performance. Recent "learned" video compression methods have explored end-to-end optimization, but most have focused on sequential (uni-directional) prediction. However, it's well-established in traditional compression that hierarchical, bi-directional prediction (using information from both past and future frames) offers superior compression efficiency. This paper addresses the gap of applying end-to-end optimization to a hierarchical, bi-directional learned video compression framework.

Methods

The proposed framework mimics traditional hierarchical video compression but replaces fixed functions with trainable deep neural networks. Key components include:

  • Image Compression Net: A CNN autoencoder, similar to those used for learned image compression, is employed to compress "key" frames (intra-coded frames). This network is based on prior work [6] but uses a Laplacian model for better performance.
  • Flow Estimation and Compression Net: Bi-directional optical flow is estimated between reference frames and the current frame. The authors leverage the SpyNet architecture [17] for this, and the estimated flow vectors are then compressed using a similar architecture to the image compression net.
  • Motion Compensation Net: A novel CNN-based motion compensation network takes warped past and future reference frames and learns to fuse them using a learned mask. This allows for sub-pixel motion compensation and aims to minimize warping artifacts, producing a refined predicted frame.
  • Residual Compression Net: Similar to the image compression net, this module compresses the residual information (the difference between the ground truth frame and the motion-compensated frame).
  • Post Processing Net: A residual block-based network is used to enhance the visual quality of reconstructed frames by reducing compression artifacts.

The entire framework is trained end-to-end using a single rate-distortion loss function that considers distortion (MSE), and estimated rates for image, flow, and residual information. Quantization is handled by adding uniform noise during training (soft quantization) and rounding during inference (hard quantization).

Datasets

The paper uses the REDS dataset [22] for training and evaluation. This dataset is known for realistic videos with complex motions, originally used for video restoration challenges. The authors downsample and crop the videos to meet network input requirements.

Key Insights and Ideas

  • Bi-directional is Better (Even for Learned Codecs): The paper empirically demonstrates that extending learned video compression to hierarchical, bi-directional prediction, mirroring the success of traditional codecs, leads to significant performance gains.
  • End-to-End Optimization of Complex Pipelines: By treating all modules (motion estimation, compensation, residual coding, etc.) as differentiable neural networks, the authors achieve a holistic optimization that is not possible with traditional methods.
  • Modular Design for Future Improvements: The framework's modularity allows for easy integration of advancements in individual components, such as improved optical flow networks or more sophisticated entropy models.
  • Learned Motion Compensation for Refinement: The novel motion compensation network that learns to fuse warped frames with a mask is a key innovation for improving prediction accuracy and reducing artifacts.

Key Results

  • Rate-Distortion Performance: The proposed method outperforms the state-of-the-art learned sequential video compression method (DVC [3]) by 0.2 to 0.5 dB across the bitrate range.
  • Comparison to Traditional Codecs: At low bitrates, the method performs comparably to x264. As bitrates increase, its performance widens the gap with x264 and approaches that of x265.
  • Visual Quality: At a bitrate of 0.14 bpp, the learned method produces smoother results than x264 and DVC, with fewer blocking artifacts, although its PSNR is slightly lower than x265.

Limitations and Future Work

  • Limitations: The paper notes that while their method approaches x265 performance, it doesn't fully surpass it. The performance is still limited by the complexity of the learned components and the optimization process.
  • Future Work: The authors suggest that their modular framework can readily incorporate future advancements in learned optical flow estimation and context modeling for entropy coding. Specifically, they mention that context models for exploiting latent distributions, as proposed in [8], could further enhance compression performance.

Summary updated

Abstract

Conventional video compression methods employ a linear transform and block motion model, and the steps of motion estimation, mode and quantization parameter selection, and entropy coding are optimized individually due to combinatorial nature of the end-to-end optimization problem. Learned video compression allows end-to-end rate-distortion optimized training of all nonlinear modules, quantization parameter and entropy model simultaneously. While previous work on learned video compression considered training a sequential video codec based on end-to-end optimization of cost averaged over pairs of successive frames, it is well-known in conventional video compression that hierarchical, bi-directional coding outperforms sequential compression. In this paper, we propose for the first time end-to-end optimization of a hierarchical, bi-directional motion compensated learned codec by accumulating cost function over fixed-size groups of pictures (GOP). Experimental results show that the rate-distortion performance of our proposed learned bi-directional GOP coder outperforms the state-of-the-art end-to-end optimized learned sequential compression as expected.
BibTeX
@inproceedings{yilmaz2020end,
  author = {Mustafa Akin Yilmaz and Ahmet Murat Tekalp},
  title = {{End-to-End Rate-Distortion Optimization for Bi-Directional Learned Video Compression}},
  booktitle = {ICIP},
  year = {2020},
  url = {https://openreview.net/forum?id=fPPdCaH2HM},
}
Figure