Abstract
Motion estimation and motion compensation are indispensable parts of inter prediction in video coding. Since the motion vector of objects is mostly in fractional pixel units, original reference pictures may not accurately provide a suitable reference for motion compensation. In this paper, we propose a deep reference picture generator which can create a picture that is more relevant to the current encoding frame, thereby further reducing temporal redundancy and improving video compression efficiency. Inspired by the recent progress of Convolutional Neural Network (CNN), this paper proposes to use a dilated CNN to build the generator. Moreover, we insert the generated deep picture into Versatile Video Coding (VVC) as a reference picture and perform a comprehensive set of experiments to evaluate the effectiveness of our network on the latest VVC Test Model VTM. The experimental results demonstrate that our proposed method achieves on average 9.7% bit saving compared with VVC under low-delay P configuration.
Introduction
Modern video compression standards like Versatile Video Coding (VVC) rely heavily on inter-prediction to reduce temporal redundancy. This process involves finding matching blocks in previously encoded frames (reference frames) to predict the current frame. However, objects in natural video rarely move in perfect integer-pixel increments. To account for fractional motion, encoders typically use fixed interpolation filters (e.g., 8-tap filters). While effective, these mathematical filters are generic and often fail to capture the complex textures and varied motion patterns of real-world scenes.
The authors propose that a more "relevant" reference picture can be generated using deep learning. By creating a reference frame that better anticipates the content of the current frame, the residual data—the difference between the prediction and the actual frame—can be significantly minimized.
Proposed Framework
The core innovation is a Deep Reference Picture Generator integrated directly into the VVC Test Model (VTM 10.0). Unlike traditional methods that rely on static interpolation, this generator uses a neural network to produce a custom reference picture for P-frames.
As shown in the framework above, the network takes a previously reconstructed frame as input and generates a "Deep Reference Picture." This generated frame is then inserted into the Reference Picture List (RPL), effectively providing the motion compensation process with a higher-quality starting point.
Dilated-Inception Architecture
To handle the multi-scale nature of video motion and textures, the researchers designed a specific Dilated-Inception block. This architecture allows the network to capture both local details and broader context without an exponential increase in parameters.
The block features:
- Multi-Scale Branches: Three branches use 3x3 convolutions with different dilation rates (1, 3, and 5). Dilated convolutions expand the "receptive field," allowing the network to see more of the surrounding image context, which is crucial for predicting motion.
- Identity Mapping: A parallel branch acts as an approximate identity mapping, ensuring that the original features of the reference frame are preserved and appropriately weighted.
- Concatenation and Fusion: The outputs of these branches are concatenated and fused via a 1x1 convolution before being added back to the input (residual learning) and passed through a ReLU activation.
Experimental Results
The proposed method was evaluated using the VVC common test sequences across various classes (WVGA to 720p). The model was trained using block pairs generated via Lucas-Kanade optical flow to simulate fractional motion.
The experimental results demonstrate significant performance gains:
- Bit Savings: The method achieved an average 9.7% reduction in BD-rate compared to the standard VTM.
- Class Performance: The highest gains were observed in Class D (15.1%) and Class C (13.3%), indicating strong performance on lower-resolution content where interpolation artifacts are often more pronounced.
- Comparison: The proposed dilated CNN outperformed other deep-learning models like VRCNN and VRF in terms of both PSNR and SSIM, confirming that the generated references are more accurate representations of the target frames.
Conclusion and Future Work
By leveraging dilated convolutions within an Inception-like framework, this research provides a powerful tool for reducing temporal redundancy in video coding, particularly for low-delay P configurations. While the method shows exceptional promise for natural video, challenges remain for screen content (Class F), which possesses distinct statistical properties. Future work aims to improve the model's transferability across diverse content types and address the computational complexity inherent in deep-learning-based video codecs.
Reference: Tian, H., Gao, P., Wei, R., & Paul, M. (2022). Dilated Convolutional Neural Network-Based Deep Reference Picture Generation for Video Compression. arXiv:2202.05514v1 [eess.IV].
Summary updated