TL;DR
This paper introduces a novel end-to-end learned video compression framework that leverages hierarchical, bi-directional motion compensation, optimizing the entire pipeline for rate-distortion performance across groups of pictures (GOPs). This approach significantly outperforms previous learned sequential video compression methods and approaches the performance of traditional codecs like x265.
What Problem Does This Paper Address?
Traditional video compression codecs (like H.264/H.265) rely on hand-engineered modules for motion estimation, transform, quantization, and entropy coding. While effective, these components are optimized individually due to the combinatorial complexity of optimizing the entire system end-to-end. This limitation prevents achieving the theoretical best rate-distortion performance. Recent "learned" video compression methods have explored end-to-end optimization, but most have focused on sequential (uni-directional) prediction. However, it's well-established in traditional compression that hierarchical, bi-directional prediction (using information from both past and future frames) offers superior compression efficiency. This paper addresses the gap of applying end-to-end optimization to a hierarchical, bi-directional learned video compression framework.
Methods
The proposed framework mimics traditional hierarchical video compression but replaces fixed functions with trainable deep neural networks. Key components include:
- Image Compression Net: A CNN autoencoder, similar to those used for learned image compression, is employed to compress "key" frames (intra-coded frames). This network is based on prior work [6] but uses a Laplacian model for better performance.
- Flow Estimation and Compression Net: Bi-directional optical flow is estimated between reference frames and the current frame. The authors leverage the SpyNet architecture [17] for this, and the estimated flow vectors are then compressed using a similar architecture to the image compression net.
- Motion Compensation Net: A novel CNN-based motion compensation network takes warped past and future reference frames and learns to fuse them using a learned mask. This allows for sub-pixel motion compensation and aims to minimize warping artifacts, producing a refined predicted frame.
- Residual Compression Net: Similar to the image compression net, this module compresses the residual information (the difference between the ground truth frame and the motion-compensated frame).
- Post Processing Net: A residual block-based network is used to enhance the visual quality of reconstructed frames by reducing compression artifacts.
The entire framework is trained end-to-end using a single rate-distortion loss function that considers distortion (MSE), and estimated rates for image, flow, and residual information. Quantization is handled by adding uniform noise during training (soft quantization) and rounding during inference (hard quantization).
Datasets
The paper uses the REDS dataset [22] for training and evaluation. This dataset is known for realistic videos with complex motions, originally used for video restoration challenges. The authors downsample and crop the videos to meet network input requirements.
Key Insights and Ideas
- Bi-directional is Better (Even for Learned Codecs): The paper empirically demonstrates that extending learned video compression to hierarchical, bi-directional prediction, mirroring the success of traditional codecs, leads to significant performance gains.
- End-to-End Optimization of Complex Pipelines: By treating all modules (motion estimation, compensation, residual coding, etc.) as differentiable neural networks, the authors achieve a holistic optimization that is not possible with traditional methods.
- Modular Design for Future Improvements: The framework's modularity allows for easy integration of advancements in individual components, such as improved optical flow networks or more sophisticated entropy models.
- Learned Motion Compensation for Refinement: The novel motion compensation network that learns to fuse warped frames with a mask is a key innovation for improving prediction accuracy and reducing artifacts.
Key Results
- Rate-Distortion Performance: The proposed method outperforms the state-of-the-art learned sequential video compression method (DVC [3]) by 0.2 to 0.5 dB across the bitrate range.
- Comparison to Traditional Codecs: At low bitrates, the method performs comparably to x264. As bitrates increase, its performance widens the gap with x264 and approaches that of x265.
- Visual Quality: At a bitrate of 0.14 bpp, the learned method produces smoother results than x264 and DVC, with fewer blocking artifacts, although its PSNR is slightly lower than x265.
Limitations and Future Work
- Limitations: The paper notes that while their method approaches x265 performance, it doesn't fully surpass it. The performance is still limited by the complexity of the learned components and the optimization process.
- Future Work: The authors suggest that their modular framework can readily incorporate future advancements in learned optical flow estimation and context modeling for entropy coding. Specifically, they mention that context models for exploiting latent distributions, as proposed in [8], could further enhance compression performance.
Summary updated