In the world of video compression, the "Holy Grail" is finding ways to represent high-quality motion with as few bits as possible. While the latest standard, Versatile Video Coding (VVC), is incredibly efficient, researchers are increasingly looking toward Deep Learning to squeeze out even more performance.
A recent paper by Haoyue Tian and colleagues, titled "Dilated Convolutional Neural Network-Based Deep Reference Picture Generation for Video Compression," proposes a clever way to improve how video encoders "predict" the next frame by generating high-quality "deep" reference pictures.
TL;DR
The authors propose a Dilated-Inception CNN that generates a custom reference picture for P-frames in VVC. By replacing standard reference frames with these AI-generated versions, they achieved an average 9.7% bit saving (BD-rate) across standard test sequences, significantly outperforming traditional interpolation methods.
What Problem Does This Paper Address?
Most video compression relies on inter-prediction: instead of saving every pixel of every frame, the encoder looks at previous frames, finds matching blocks, and only records the "motion vector" (where the block moved) and the "residual" (the tiny differences between the blocks).
The catch? Objects rarely move in perfect integer-pixel increments. They move by fractions of a pixel. To handle this, encoders use fixed mathematical filters (like 8-tap filters) to interpolate "sub-pixels." However, these fixed filters are "one-size-fits-all" and often fail to capture the complex textures and varied motion found in natural video. The authors argue that if we can generate a reference picture that is more "relevant" to the current frame than a simple interpolated one, we can drastically reduce the amount of data needed to describe the differences.
Methods
The core of the paper is a Deep Reference Picture Generator integrated directly into the VVC Test Model (VTM 10.0).
The Architecture: Dilated-Inception Blocks
The researchers didn't just use a standard CNN. They designed a specific Dilated-Inception block to handle the multi-scale nature of video motion:
- Multi-Scale Branches: The block splits into three branches with different "dilation rates" (1, 3, and 5). Dilated convolutions allow the network to have a much larger "receptive field" (seeing more of the image context) without increasing the number of parameters or losing resolution.
- Identity Mapping: A fourth branch acts as an approximate identity branch, ensuring that the original features of the reference frame are preserved and weighted appropriately.
- Integration: The network takes the previous reconstructed frame as input and outputs a "Deep Reference Picture." This picture is then inserted into the Reference Picture List (RPL), replacing the first original reference picture.
Training Strategy
The authors used a block-based training approach. They used the Lucas-Kanade optical flow method to calculate fractional motion vectors to create their ground-truth training pairs. Interestingly, they tested various block sizes and found that 32x32 blocks provided the best balance between local detail and global context.
Datasets
- Training: The model was trained using the
BlowingBubblessequence, generating over 42,000 block pairs. - Evaluation: The system was tested on the standard HEVC/VVC common test sequences, categorized into Classes C (WVGA), D (WQVGA), E (720p), and F (Screen Content).
Key Insights and Ideas
- Context is King: By using dilated convolutions, the network can "see" surrounding pixels to better predict how a texture should look after moving, something a local 8-tap filter simply cannot do.
- P-Frame Focus: While previous work often focused on B-frames (which have two reference frames to interpolate between), this paper tackles the harder task of P-frames (low-delay P configuration), where only one previous frame is available.
- Efficiency at Low Bitrates: The results show that the proposed method performs exceptionally well at higher Quantization Parameters (QPs), meaning it provides the most benefit when bandwidth is most limited.
Key Results
The proposed method was compared against the standard VTM and two other deep-learning models (VRCNN and VRF):
- Significant Bit Savings: An average 9.7% reduction in BD-rate across all classes.
- Class-Specific Wins: The gains were particularly high in Class D (15.1%) and Class C (13.3%).
- Visual Quality: PSNR and SSIM metrics for the generated deep references were consistently higher than competing models, indicating the generated frames were much closer to the actual "ground truth" current frame.
Limitations and Future Work
- Screen Content Challenge: The model performed poorly on "Class F" (screen content like text and UI). The authors note that screen content has very different motion and texture characteristics than natural video, and their model (trained on natural video) didn't transfer well to this domain.
- Computational Complexity: Like most CNN-based compression tools, the primary hurdle for real-world adoption is the encoding/decoding time. Running a deep neural network for every reference frame adds significant overhead.
- Future Directions: The authors suggest that future work will focus on improving transfer performance—making the model more robust across different types of content without needing specific retraining.
Reference: Tian, H., Gao, P., Wei, R., & Paul, M. (2022). Dilated Convolutional Neural Network-Based Deep Reference Picture Generation for Video Compression. arXiv:2202.05514v1 [eess.IV].
Summary updated