Overview
Neural video compression (NVC) has seen rapid advancement, yet many existing frameworks rely on a uni-directional information flow where motion vectors are simply passed to a frame coder. This paper introduces Motion Information Propagation (MIP), a framework that establishes a bi-directional feedback loop between motion and frame coding. By sharing high-dimensional features, the proposed DCVC-MIP achieves a 12.9% bitrate saving over previous state-of-the-art neural codecs.
The Challenge: Uni-directional Information Flow
Traditional and many neural video codecs utilize motion compensation to exploit temporal correlation. In these systems, motion estimation produces vectors that are coded and then used to warp previous frames to predict the current one. However, this "one-way street" approach has several drawbacks:
- Alignment Errors: Simple 2D motion vectors often struggle to align complex textures, leading to artifacts like ghosting or blur.
- Isolated Motion Coding: Motion information is often encoded without leveraging the rich contextual features already decoded in the frame buffer.
- Short-term Memory: Maintaining long-range temporal consistency is difficult without a mechanism to propagate information across multiple steps.
Proposed Method: Motion Information Propagation (MIP)
The core of the DCVC-MIP framework is the synergy created between motion and frame coding through two primary paths:
1. Motion Guidance (The Blue Path)
High-dimensional features from the motion decoder are integrated into the context generation module. These features act as "guidance" for alignment, helping the system resolve ambiguities that standard optical flow might miss, particularly in high-texture areas.
2. Motion Condition (The Red Path)
This feedback loop propagates features from the frame's context generation forward to serve as a condition for coding the subsequent frame's motion. By utilizing the scene's context, the codec can represent future motion vectors more efficiently, reducing the motion coding bitrate by approximately 24%.
Hybrid Context Generation
To handle diverse motion types, the authors implement a multi-scale context generation strategy:
- High-Resolution Branch: Uses "offset diversity" to predict multiple sub-pixel offsets and masks, ensuring sharp reconstruction of fine details.
- Low-Resolution Branch: Employs a Swin Transformer-based refinement module to model long-range dependencies, effectively handling occlusions and complex movements.
Experimental Results
The model was trained on the Vimeo-90k dataset and evaluated against benchmarks including UVG, MCL-JCV, and HEVC Common Test Conditions.
Performance Metrics
- Bitrate Savings: DCVC-MIP achieves a 12.9% average bitrate reduction compared to the previous state-of-the-art, DCVC-HEM.
- Comparison to Standards: It outperforms the VTM-13.2 (H.266/VVC) reference software by 16.6% in terms of PSNR.
- Visual Fidelity: Qualitative analysis shows superior handling of high-texture regions (e.g., fine details on objects) and reduced artifacts in areas of large movement where traditional warping often fails.
Conclusion
By transforming the relationship between motion and frame coding into a collaborative interaction, Motion Information Propagation significantly enhances the efficiency of neural video compression. While the computational complexity of Transformers remains a consideration for real-time applications, the substantial gains in rate-distortion performance mark a significant step forward for the field.
Summary updated