Dynamically Expanded CNN Array for Video Coding

Everett Fall, Kai-Wei Chang, Liang-Gee Chen

Abstract

Video coding is a critical step in all popular methods of streaming video. Marked progress has been made in video quality, compression, and computational efficiency. Recently, there has been an interest in finding ways to apply techniques from the fast-progressing field of machine learning to further improve video coding. We present a method that uses convolutional neural networks to help refine the output of various standard coding methods. The novelty of our approach is to train multiple different sets of network parameters, with each set corresponding to a specific, short segment of video. The array of network parameter sets expands dynamically to match a video of any length. We show that our method can improve the quality and compression efficiency of standard video codecs.

Introduction

Traditional video codecs, such as H.264/AVC and H.265/HEVC, form the backbone of modern digital video streaming. These codecs rely on block-based compression techniques which, while efficient, often introduce visible "blocky" artifacts, particularly at lower bitrates. While machine learning (ML) offers a promising avenue for mitigating these artifacts, two significant challenges persist: the high computational complexity of replacing optimized codecs with end-to-end neural networks, and the difficulty of a single generic model generalizing across diverse video content.

This paper proposes a hybrid "post-processing" approach. Instead of replacing the codec, we utilize a dynamic array of lightweight, specialized Convolutional Neural Networks (CNNs) to refine the codec's output. By training unique parameters for short video segments, the system can surgically remove compression artifacts and restore fine-grained textures.

Methodology: Dynamically Expanded CNN Array

The core innovation of this work is the use of a dynamically expanding array of specialized CNN parameters. Rather than employing a single global network, the video is partitioned into short segments (e.g., 50 frames each).

Segment-Specific Training and Refinement

For each segment, a specific set of network parameters ($\theta$) is learned. This allows the CNN to specialize in the specific textures and motion patterns of that short interval. The refinement module $R$ takes the output of a standard codec (e.g., H.264) and applies the specialized CNN to produce a higher-quality frame $X'$. The network can be trained to predict the original frame directly or to learn the "residual"—the difference between the original and the compressed version.

The proposed encoding and refinement scheme, showing the interaction between the standard codec, the parameter selector, and the refinement module R.

Dynamic Expansion and Random Access

As the video length increases, the array of parameter sets grows accordingly. During decoding, the system simply switches to the appropriate parameter set for the current segment. This architecture is inherently compatible with random access; a decoder only needs the specific parameters for a given segment to begin high-quality playback, making it practical for streaming applications.

Experimental Results

The method was evaluated using the "Big Buck Bunny" benchmark dataset. A baseline was established using the H.264 codec at a high Constant Rate Factor (CRF 36) to simulate low-quality, high-compression scenarios.

Quantitative and Qualitative Performance

The experimental results demonstrate a substantial improvement in both objective metrics and subjective visual quality. The Peak Signal-to-Noise Ratio (PSNR) increased from a baseline of 33.82 dB to 36.96 dB after refinement. Similarly, the Multi-Scale Structural Similarity (MS-SSIM) index improved from 0.9589 to 0.9802.

Visual comparison of the original frame (left), the compressed H.264 output (middle), and the refined output using the specialized CNN array (right).

Visually, the refiner significantly reduces blocky artifacts, bringing the output much closer to the ground truth original frames. This confirms that a small network specialized for a specific scene can outperform larger, more generalized models.

Conclusion and Future Work

This paper presents a robust hybrid approach to video coding that leverages the strengths of traditional codecs for motion compensation and specialized CNNs for texture reconstruction. While the results are promising, future work will address the computational overhead of training parameters for every segment and investigate the bitrate overhead required to transmit these parameters. Additionally, testing the method against more modern codecs like AV1 or VVC remains a key area for further research.

Summary updated

Abstract

Video coding is a critical step in all popular methods of streaming video. Marked progress has been made in video quality, compression, and computational efficiency. Recently, there has been an interest in finding ways to apply techniques form the fast-progressing field of machine learning to further improve video coding. We present a method that uses convolutional neural networks to help refine the output of various standard coding methods. The novelty of our approach is to train multiple different sets of network parameters, with each set corresponding to a specific, short segment of video. The array of network parameter sets expands dynamically to match a video of any length. We show that our method can improve the quality and compression efficiency of standard video codecs.
BibTeX
@inproceedings{fall2020dynamically,
  author = {Everett Fall and Kai-Wei Chang and Liang-Gee Chen},
  title = {{Dynamically Expanded CNN Array for Video Coding}},
  booktitle = {ICIGP},
  year = {2020},
  url = {https://openreview.net/forum?id=4j74ttQe3K},
}
Figure