Dynamically Expanded CNN Array for Video Coding

Everett Fall, Kai-Wei Chang, Liang-Gee Chen

TL;DR

Instead of replacing traditional video codecs with massive AI models, this paper proposes a "post-processing" approach using an array of tiny, specialized Convolutional Neural Networks (CNNs). By training a unique set of parameters for every short segment of a video, the system can surgically remove compression artifacts, significantly boosting visual quality with minimal overhead.

What Problem Does This Paper Address?

Traditional video codecs like H.264/AVC and H.265/HEVC are the backbone of modern streaming. However, they rely on block-based compression, which often results in "blocky" artifacts, especially at low bitrates.

While Machine Learning (ML) offers a promising way to fix these issues, two major hurdles remain:

  1. Complexity: Replacing a highly optimized codec like H.265 with an end-to-end neural network is computationally expensive and difficult to optimize.
  2. Generalization: A single neural network trained on millions of generic images might struggle to perfectly refine a specific, unique scene in a movie.

The authors tackle these challenges by treating ML as a "refiner" rather than a replacement, and by moving away from the "one-size-fits-all" model.

Methods

The core innovation is the Dynamically Expanded CNN Array. Rather than using one large network for an entire video, the authors partition the video into short segments (e.g., 50 frames each).

  1. Segment-Specific Training: For every segment, the system learns a specific set of network parameters ($\theta$). This allows the CNN to "overfit" or specialize in the textures and motions of just those few seconds of video.
  2. Refinement Module: The refiner takes the output of a standard codec (like H.264) and applies the specialized CNN to produce a higher-quality frame. It can either predict the original frame directly or learn to predict the "residual" (the difference between the original and the compressed version).
  3. Dynamic Expansion: As the video gets longer, the array of parameter sets grows. During playback, the decoder simply switches to the correct set of parameters for the current segment.
  4. Architecture: The refiner is relatively lightweight, consisting of four convolutional layers with 5x5 filters followed by three layers with 3x3 filters.

Datasets

The authors evaluated their method using the "Big Buck Bunny" benchmark dataset, a standard in the video processing community. They used the H.264 codec at a high Constant Rate Factor (CRF 36) to create a low-quality, high-compression baseline for the refiner to improve upon.

Key Insights and Ideas

  • Specialization over Generalization: The "aha moment" here is that a small network that knows a specific scene very well can outperform a massive network that knows "all" scenes. By training on 50-frame segments, the network captures local temporal nuances that a global model would miss.
  • Compatibility with Random Access: Video streaming requires the ability to jump to any point in a video (random access). Because each segment has its own parameters, the decoder only needs the specific parameters for that segment to start refining, making it highly practical for streaming.
  • Hybrid Approach: By keeping the traditional codec as the base, the system benefits from decades of optimization in motion compensation while using AI only for what it does best: fine-grained texture reconstruction.

Key Results

The experimental results showed a "substantial improvement" in both objective and subjective quality:

  • PSNR (Peak Signal-to-Noise Ratio): Increased from 33.82 dB (original H.264) to 36.96 dB (refined).
  • MS-SSIM (Multi-Scale Structural Similarity): Improved from 0.9589 to 0.9802.

Visually, the refiner was able to significantly reduce the blocky artifacts typical of high-compression H.264, bringing the output much closer to the "ground truth" original frames.

Limitations and Future Work

While the paper demonstrates a strong proof-of-concept, a few areas remain for further exploration:

  • Computational Overhead: Training a new set of parameters for every 50 frames of a long movie is computationally intensive. The authors mention that training took about 500 steps per segment, which might be slow for real-time encoding.
  • Metadata Size: While the authors claim the increase in "code size" is minimal, the paper does not provide a detailed breakdown of the bitrate overhead required to transmit the CNN parameters ($\theta$) alongside the video data.
  • Scaling: Future work could explore how this method performs on more modern codecs like AV1 or VVC, and whether the segment size ($\rho$) can be optimized dynamically based on scene complexity.

Summary updated

Abstract

Video coding is a critical step in all popular methods of streaming video. Marked progress has been made in video quality, compression, and computational efficiency. Recently, there has been an interest in finding ways to apply techniques from the fast-progressing field of Machine Learning to further improve video coding.We present a method that uses convolutional neural networks to help refine the output of various standard coding methods. The novelty of our approach is to train multiple different groups of network parameters, with each set corresponding to a specific, short segment of video and arranging the groups in a hierarchy that reflects their locality within the video. Low-level groups are updated often and specialize on local features while high-level groups find non-local features that can be used for longer segments of video. The parameter groups expand dynamically to match a video of any length. We show that our method can improve the quality of standard video codecs without increasing in compressed video size.
BibTeX
@inproceedings{fall2020dynamically,
  author = {Everett Fall and Kai-Wei Chang and Liang-Gee Chen},
  title = {{Dynamically Expanded CNN Array for Video Coding}},
  booktitle = {ICIGP},
  year = {2020},
  url = {https://openreview.net/forum?id=4j74ttQe3K},
}

Loading related content...

Figure