Video Compression Artifact Reduction by Fusing Motion Compensation and Global Context in a Swin-CNN Based Parallel Architecture

Xinjian Zhang, Su Yang, Wuyang Luo, Longwen Gao, Weishan ZhangAAAI 2023 · DOI

TL;DR

This paper introduces a novel framework, STCF, that fuses motion compensation and global context to significantly improve video compression artifact reduction. By leveraging a Swin-CNN architecture, it achieves state-of-the-art performance with improved efficiency and detail restoration.

What Problem Does This Paper Address?

Video streaming and online video platforms are booming, leading to a massive increase in video data traffic. To manage storage and bandwidth constraints, lossy video compression techniques like H.264/AVC and H.265/HEVC are widely employed. However, these compression methods introduce undesirable artifacts such as blocking, blurring, and ringing, which degrade the visual quality and negatively impact downstream video analysis tasks. While existing methods focus on either motion compensation or global context, they often fail to comprehensively utilize the redundant information available in compressed videos, leading to suboptimal artifact reduction. This paper aims to bridge this gap by proposing a unified approach that effectively combines both motion compensation and global context.

Methods

The core of this work is the Spatio-Temporal Compensation Fusion (STCF) framework. It comprises two main modules:

  • Spatiotemporal Alignment (SA) Module: This module is responsible for aligning the motion of input frames. It takes a target frame and its surrounding reference frames as input and outputs a shallow temporal feature. The architecture of this module is similar to that used in prior works like STDF.
  • Quality Enhancement (QE) Module: This module takes the shallow temporal feature from the SA module and the compressed target frame to infer the high-quality enhanced frame. The key innovation here is the Parallel Swin-CNN Fusion (PSCF) block.
  • Motion Compensation: Within the PSCF block, motion compensation is extracted using an Ada-CNN layer. This layer is an enhanced residual block incorporating wide activation and adaptive parameters for better feature extraction.
  • Global Context: Global context is captured using a Swin-based temporal self-attention mechanism. This approach adapts the Swin Transformer's shifted window strategy to video, significantly reducing the computational complexity compared to standard self-attention for high-resolution videos.
  • Fusion: The motion compensation and global context features are then fused within the PSCF block. The paper explores different fusion schemes, ultimately selecting a "Concat & Conv" approach. The QE module stacks multiple PSCF blocks to iteratively refine the compensation information.

To further optimize computational efficiency, a "Down Head" is used to reduce spatial resolution before the PSCF blocks, and an "Up Tail" restores it afterward.

Datasets

The experiments and evaluations were conducted on the MFQEv2 dataset. This dataset comprises 126 video sequences collected from various sources (Xiph, VQEG2, and JCT-VC) with diverse resolutions and contents. For training, 108 sequences were used, and the remaining 18 were reserved for testing. The videos were compressed using HM 16.20 with the HEVC Low-Delay-P (LDP) configuration at five different Quantization Parameters (QPs): 22, 27, 32, 37, and 42.

Key Insights and Ideas

  • Synergy of Motion and Context: The paper strongly advocates for the complementary nature of motion compensation and global context. By fusing them, the framework can leverage richer information than either approach alone, leading to superior artifact reduction.
  • Efficient Global Context Capture: The adaptation of the Swin Transformer's shifted window mechanism to temporal self-attention is a crucial innovation. It makes capturing long-range temporal dependencies computationally feasible for high-resolution video processing, overcoming a major bottleneck in previous methods.
  • Ada-CNN for Motion: The proposed Ada-CNN layer provides an effective way to extract deep spatiotemporal information for motion compensation, building upon residual learning principles.
  • Parallel Fusion for Enhanced Refinement: The PSCF block, by parallelizing motion compensation and global context extraction and then fusing them, allows for iterative refinement of the enhanced frame, leading to better detail restoration.

Key Results

  • State-of-the-Art Performance: The proposed STCF framework achieves state-of-the-art performance on the MFQEv2 dataset, outperforming existing methods by up to 0.23dB (27% improvement) in PSNR compared to RFDA.
  • Improved Detail Restoration: Qualitative results show that the STCF framework restores more details and textures in the enhanced frames compared to other methods, which tend to over-smooth the images.
  • Stable Quality: The method demonstrates the most stable quality enhancement, evidenced by the smallest averaged Peak-Valley Difference (PVD) and Standard Deviation (SD) across test sequences.
  • Rate-Distortion Performance: The method shows better rate-distortion performance, achieving higher PSNR at similar bitrates.
  • Efficiency: While the method is computationally intensive due to fusion, it achieves a good trade-off between performance and efficiency, with a smaller parameter count than some comparable methods.

Limitations and Future Work

  • Computational Cost: Although the Swin-based attention reduces complexity, the fusion of motion compensation and global context still results in a higher computational cost compared to methods that only use one. The paper notes that their method has a higher run-time than some baselines, though it offers better performance and fewer parameters than BasicVSR++.
  • Future Work: The authors suggest that the PSCF block, designed for fusing motion compensation and global context, can be easily adapted to other video low-level tasks. They plan to extend their method to explore other such tasks.

Summary updated

BibTeX
@inproceedings{zhang2023video,
  author = {Xinjian Zhang and Su Yang and Wuyang Luo and Longwen Gao and Weishan Zhang},
  title = {{Video Compression Artifact Reduction by Fusing Motion Compensation and Global Context in a Swin-CNN Based Parallel Architecture}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25458},
}
Figure