Dilated Convolutional Neural Network-Based Deep Reference Picture Generation for Video Compression

Haoyue Tian, Pan Gao, Ran Wei, Manoranjan Paul

In the world of video compression, the "Holy Grail" is finding ways to represent high-quality motion with as few bits as possible. While the latest standard, Versatile Video Coding (VVC), is incredibly efficient, researchers are increasingly looking toward Deep Learning to squeeze out even more performance.

A recent paper by Haoyue Tian and colleagues, titled "Dilated Convolutional Neural Network-Based Deep Reference Picture Generation for Video Compression," proposes a clever way to improve how video encoders "predict" the next frame by generating high-quality "deep" reference pictures.

TL;DR

The authors propose a Dilated-Inception CNN that generates a custom reference picture for P-frames in VVC. By replacing standard reference frames with these AI-generated versions, they achieved an average 9.7% bit saving (BD-rate) across standard test sequences, significantly outperforming traditional interpolation methods.

What Problem Does This Paper Address?

Most video compression relies on inter-prediction: instead of saving every pixel of every frame, the encoder looks at previous frames, finds matching blocks, and only records the "motion vector" (where the block moved) and the "residual" (the tiny differences between the blocks).

The catch? Objects rarely move in perfect integer-pixel increments. They move by fractions of a pixel. To handle this, encoders use fixed mathematical filters (like 8-tap filters) to interpolate "sub-pixels." However, these fixed filters are "one-size-fits-all" and often fail to capture the complex textures and varied motion found in natural video. The authors argue that if we can generate a reference picture that is more "relevant" to the current frame than a simple interpolated one, we can drastically reduce the amount of data needed to describe the differences.

Methods

The core of the paper is a Deep Reference Picture Generator integrated directly into the VVC Test Model (VTM 10.0).

The Architecture: Dilated-Inception Blocks

The researchers didn't just use a standard CNN. They designed a specific Dilated-Inception block to handle the multi-scale nature of video motion:

  • Multi-Scale Branches: The block splits into three branches with different "dilation rates" (1, 3, and 5). Dilated convolutions allow the network to have a much larger "receptive field" (seeing more of the image context) without increasing the number of parameters or losing resolution.
  • Identity Mapping: A fourth branch acts as an approximate identity branch, ensuring that the original features of the reference frame are preserved and weighted appropriately.
  • Integration: The network takes the previous reconstructed frame as input and outputs a "Deep Reference Picture." This picture is then inserted into the Reference Picture List (RPL), replacing the first original reference picture.

Training Strategy

The authors used a block-based training approach. They used the Lucas-Kanade optical flow method to calculate fractional motion vectors to create their ground-truth training pairs. Interestingly, they tested various block sizes and found that 32x32 blocks provided the best balance between local detail and global context.

Datasets

  • Training: The model was trained using the BlowingBubbles sequence, generating over 42,000 block pairs.
  • Evaluation: The system was tested on the standard HEVC/VVC common test sequences, categorized into Classes C (WVGA), D (WQVGA), E (720p), and F (Screen Content).

Key Insights and Ideas

  • Context is King: By using dilated convolutions, the network can "see" surrounding pixels to better predict how a texture should look after moving, something a local 8-tap filter simply cannot do.
  • P-Frame Focus: While previous work often focused on B-frames (which have two reference frames to interpolate between), this paper tackles the harder task of P-frames (low-delay P configuration), where only one previous frame is available.
  • Efficiency at Low Bitrates: The results show that the proposed method performs exceptionally well at higher Quantization Parameters (QPs), meaning it provides the most benefit when bandwidth is most limited.

Key Results

The proposed method was compared against the standard VTM and two other deep-learning models (VRCNN and VRF):

  • Significant Bit Savings: An average 9.7% reduction in BD-rate across all classes.
  • Class-Specific Wins: The gains were particularly high in Class D (15.1%) and Class C (13.3%).
  • Visual Quality: PSNR and SSIM metrics for the generated deep references were consistently higher than competing models, indicating the generated frames were much closer to the actual "ground truth" current frame.

Limitations and Future Work

  • Screen Content Challenge: The model performed poorly on "Class F" (screen content like text and UI). The authors note that screen content has very different motion and texture characteristics than natural video, and their model (trained on natural video) didn't transfer well to this domain.
  • Computational Complexity: Like most CNN-based compression tools, the primary hurdle for real-world adoption is the encoding/decoding time. Running a deep neural network for every reference frame adds significant overhead.
  • Future Directions: The authors suggest that future work will focus on improving transfer performance—making the model more robust across different types of content without needing specific retraining.

Reference: Tian, H., Gao, P., Wei, R., & Paul, M. (2022). Dilated Convolutional Neural Network-Based Deep Reference Picture Generation for Video Compression. arXiv:2202.05514v1 [eess.IV].

Summary updated

Abstract

Motion estimation and motion compensation are indispensable parts of inter prediction in video coding. Since the motion vector of objects is mostly in fractional pixel units, original reference pictures may not accurately provide a suitable reference for motion compensation. In this paper, we propose a deep reference picture generator which can create a picture that is more relevant to the cur-rent encoding frame, thereby further reducing temporal redundancy and improving video compression efficiency. Inspired by the recent progress of Convolutional Neural Network(CNN), this paper pro-poses to use a dilated CNN to build the generator. Moreover, we insert the generated deep picture into Versatile Video Coding(VVC) as a reference picture and perform a comprehensive set of experiments to evaluate the effectiveness of our network on the latest VVC Test Model–VTM. The experimental results demonstrate that our pro-posed method achieves on average 9.7% bit saving compared with VVC under low-delay P configuration.
BibTeX
@inproceedings{tian2022dilated,
  author = {Haoyue Tian and Pan Gao and Ran Wei and Manoranjan Paul},
  title = {{Dilated Convolutional Neural Network-Based Deep Reference Picture Generation for Video Compression}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=wOB0LRjpGki},
}

Loading related content...

Figure