Abstract
Block-based video codecs such as Versatile Video Coding (VVC)/H.266, High Efficiency Video Coding (HEVC)/H.265, and Advanced Video Coding (AVC)/H.264 inherently introduce compression artifacts. Although these codecs have in-loop filters to correct these distortions, they are not always effective due to the complexity of the noise. Recently, deep-learning approaches emerged as a promising solution for in-loop filtering. However, most previous approaches were designed solely for learning from images and neglected the high-frequency signals present in the reconstructed video frames. In this article, we propose a novel framework called TSF-Net, which jointly learns from both the pixel (spatial) and frequency-decomposed information and, through the powerful capability of a channel-wise transformer, fuses both types of information to improve performance. Our experimental results show that the proposed TSF-Net achieves a Bjøntegaard Delta (BD)-bitrate saving of up to 10.258% for the luma (Y) component under the all-intra (AI) profile, outperforming the VVC baseline and other state-of-the-art methods. Moreover, the proposed TSF-Net with an efficient channel-wise transformer is twice as efficient as TSF-Net with a vanilla channel-wise transformer.
Introduction
Modern video compression standards like VVC/H.266 utilize block-based operations that, while efficient for storage and transmission, often result in visual artifacts such as blocking, ringing, and blurring. Standard in-loop filters like the Deblocking Filter (DBF), Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF) are designed to mitigate these issues but often fall short when faced with complex noise patterns.
While deep learning has introduced more robust filtering techniques, two significant gaps remain: the neglect of frequency-domain data and the use of inefficient feature fusion methods. TSF-Net addresses these by incorporating frequency priors and utilizing an efficient transformer-based fusion mechanism.
TSF-Net Architecture
TSF-Net (Transformer-based Spatial and Frequency-decomposed Feature Fusion Network) is integrated directly into the VVC codec as an in-loop filter. It processes the reconstructed frame before it is stored in the Decoded Picture Buffer (DPB).
The network architecture consists of three primary stages:
- The Head: This stage accepts three spatial inputs—the reconstructed image, the residue, and partition information—along with two frequency-decomposed inputs. It employs "pixel-unshuffling" to reduce spatial resolution while increasing channel depth, facilitating more efficient processing.
- The Body (Residual Fusion Blocks): The core of TSF-Net comprises 20 cascaded Residual Fusion Blocks (RFBs). Each block processes spatial and frequency features in parallel, fusing them via a specialized transformer layer.
- The Tail: A final convolution and pixel-shuffle layer reconstruct the filtered image, with a global skip connection ensuring the preservation of original input details.
Joint Spatial and Frequency Learning
A key innovation of TSF-Net is the use of frequency-decomposed information as a prior. By providing the network with Discrete Cosine Transform (DCT) coefficients, the model can better distinguish between high-frequency image details and high-frequency quantization noise.
This dual-domain approach allows the model to preserve textures that traditional filters might otherwise blur, leveraging the spectral characteristics of the compressed signal. By treating each channel as a token (spectral attention), TSF-Net learns which specific frequency bands or spatial features are most important for a given block.
Efficient Transformer-Based Fusion
To manage the computational complexity typically associated with transformers, TSF-Net introduces Efficient Spectral-wise Multi-head Self-attention (E-SMSA). Unlike standard spatial attention, this mechanism treats each channel as a token. By applying average pooling to the "Key" and "Query" branches, the authors reduced the attention mechanism's complexity by 75% without sacrificing restoration quality. This modification makes TSF-Net twice as efficient as a vanilla transformer implementation, reducing inference time from 0.051s to 0.026s.
Experimental Results
The model was trained on the Div2K dataset and evaluated using 19 JVET common test sequences across various resolutions, from Class A1 (4K) to Class D.
- Bitrate Savings: TSF-Net achieved an average BD-rate saving of 10.258% for the luma (Y) component under the All-Intra (AI) profile, significantly outperforming state-of-the-art models like DAM and Uformer.
- Visual Quality: Qualitative analysis indicates that TSF-Net effectively reconstructs missing details suppressed during compression, such as reflections and fine textures, rather than merely blurring the noise.
Conclusion and Future Work
TSF-Net represents a significant step forward in learned in-loop filtering by effectively combining spatial and frequency features through an efficient transformer architecture. While currently focused on the Luma component and the All-Intra profile, future work will extend the model to Chroma components and temporal configurations like Random Access (RA) and Low Delay (LD). Additionally, the authors aim to explore scaling the transformer architecture for ultra-high resolutions where long-range correlations are even more critical.
Summary updated