Abstract
Convolutional Neural Network (CNN) and Transformer have attracted much attention recently for video post-processing (VPP). However, the interaction between CNN and Transformer in existing VPP methods is not fully explored, leading to inefficient communication between the local and global extracted features. In this paper, we explore the interaction between CNN and Transformer in the task of VPP, and propose a novel Spatial and Channel Hybrid-Attention Video Post-Processing Network (SC-HVPPNet), which can cooperatively exploit the image priors in both spatial and channel domains. Specifically, in the spatial domain, a novel spatial attention fusion module is designed, in which two attention weights are generated to fuse the local and global representations collaboratively. In the channel domain, a novel channel attention fusion module is developed, which can blend the deep representations at the channel dimension dynamically. Extensive experiments show that SC-HVPPNet notably boosts video restoration quality, with average bitrate savings of 5.29%, 12.42%, and 13.09% for Y, U, and V components in the VTM-11.0-NNVC RA configuration.
Introduction
Video compression is a fundamental technology for modern digital media, balancing file size with visual quality. While standards like Versatile Video Coding (VVC/H.266) offer high efficiency, low-bitrate compression often introduces artifacts such as blocking and ringing. Video Post-Processing (VPP) networks are employed to mitigate these issues.
Currently, VPP research is divided into two primary approaches: CNN-based methods, which excel at capturing local textures, and Transformer-based methods, which are superior at modeling global context. However, existing models often fail to efficiently integrate these two types of features. SC-HVPPNet addresses this by introducing a sophisticated hybrid-attention mechanism that bridges the gap between local detail and global dependencies.
Proposed Architecture: SC-HVPPNet
The core innovation of SC-HVPPNet is the Hybrid Fusion Block (HFB), which facilitates the interaction between local (CNN) and global (Transformer) features. The network follows a residual learning framework, predicting and subtracting compression artifacts from the input frame.
Key modules within the architecture include:
- Feature Extraction Modules (FEM): The network splits the input into a Local FEM (LFEM), using cascaded convolutions, and a Global FEM (GFEM), utilizing Swin-Transformer blocks.
- Spatial Attention Fusion Module (SAFM): This module calculates spatial attention weights to prioritize important regions across both local and global paths using a polarized self-attention structure.
- Channel Attention Fusion Module (CAFM): This dynamically blends representations across the channel dimension, ensuring the network emphasizes the most informative feature maps regardless of their origin path.
- Hybrid-Attention Fusion Module (HAFM): This serves as the integration point where SAFM and CAFM outputs are combined to produce a unified, high-quality feature map.
Experimental Results
The model was trained on the BVI-DVC dataset and evaluated using the industry-standard JVET-CTC sequences. Performance was measured using BD-rate, representing the bitrate savings required to achieve visual quality equivalent to the VTM-11.0-NNVC baseline.
SC-HVPPNet achieved significant average bitrate savings:
- Y (Luminance): 5.29%
- U (Chroma): 12.42%
- V (Chroma): 13.09%
Qualitative results demonstrate that the hybrid approach is particularly effective at restoring chroma components, which are often heavily degraded in standard compression. The model consistently outperformed several state-of-the-art VPP methods, especially in lower-resolution video categories.
Conclusion and Future Work
SC-HVPPNet demonstrates that a collaborative fusion of CNN and Transformer features via dual spatial and channel attention significantly improves video restoration. While the current spatial-only approach yields impressive gains, future research will explore extending this hybrid-attention mechanism into the temporal domain to further enhance consistency across video frames. Additionally, optimizing the computational complexity of the Transformer components remains a priority for real-time application.
Summary updated