Joint Pixel and Frequency Feature Learning and Fusion via Channel-Wise Transformer for High-Efficiency Learned In-Loop Filter in VVC

Birendra Kathariya, Zhu Li, Geert Van der Auwera

Abstract

Block-based video codecs such as Versatile Video Coding (VVC)/H.266, High Efficiency Video Coding (HEVC)/H.265, and Advanced Video Coding (AVC)/H.264 inherently introduce compression artifacts. Although these codecs have in-loop filters to correct these distortions, they are not always effective due to the complexity of the noise. Recently, deep-learning approaches emerged as a promising solution for in-loop filtering. However, most previous approaches were designed solely for learning from images and neglected the high-frequency signals present in the reconstructed video frames. In this article, we propose a novel framework called TSF-Net, which jointly learns from both the pixel (spatial) and frequency-decomposed information and, through the powerful capability of a channel-wise transformer, fuses both types of information to improve performance. Our experimental results show that the proposed TSF-Net achieves a Bjøntegaard Delta (BD)-bitrate saving of up to 10.258% for the luma (Y) component under the all-intra (AI) profile, outperforming the VVC baseline and other state-of-the-art methods. Moreover, the proposed TSF-Net with an efficient channel-wise transformer is twice as efficient as TSF-Net with a vanilla channel-wise transformer.

Introduction

Modern video compression standards like VVC/H.266 utilize block-based operations that, while efficient for storage and transmission, often result in visual artifacts such as blocking, ringing, and blurring. Standard in-loop filters like the Deblocking Filter (DBF), Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF) are designed to mitigate these issues but often fall short when faced with complex noise patterns.

While deep learning has introduced more robust filtering techniques, two significant gaps remain: the neglect of frequency-domain data and the use of inefficient feature fusion methods. TSF-Net addresses these by incorporating frequency priors and utilizing an efficient transformer-based fusion mechanism.

TSF-Net Architecture

TSF-Net (Transformer-based Spatial and Frequency-decomposed Feature Fusion Network) is integrated directly into the VVC codec as an in-loop filter. It processes the reconstructed frame before it is stored in the Decoded Picture Buffer (DPB).

TSF-Net integrated as a learned in-loop filter within the VVC framework.

The network architecture consists of three primary stages:

  1. The Head: This stage accepts three spatial inputs—the reconstructed image, the residue, and partition information—along with two frequency-decomposed inputs. It employs "pixel-unshuffling" to reduce spatial resolution while increasing channel depth, facilitating more efficient processing.
  2. The Body (Residual Fusion Blocks): The core of TSF-Net comprises 20 cascaded Residual Fusion Blocks (RFBs). Each block processes spatial and frequency features in parallel, fusing them via a specialized transformer layer.
  3. The Tail: A final convolution and pixel-shuffle layer reconstruct the filtered image, with a global skip connection ensuring the preservation of original input details.

Joint Spatial and Frequency Learning

A key innovation of TSF-Net is the use of frequency-decomposed information as a prior. By providing the network with Discrete Cosine Transform (DCT) coefficients, the model can better distinguish between high-frequency image details and high-frequency quantization noise.

The frequency decomposition process using 4x4 DCT and pixel unshuffling.

This dual-domain approach allows the model to preserve textures that traditional filters might otherwise blur, leveraging the spectral characteristics of the compressed signal. By treating each channel as a token (spectral attention), TSF-Net learns which specific frequency bands or spatial features are most important for a given block.

Efficient Transformer-Based Fusion

To manage the computational complexity typically associated with transformers, TSF-Net introduces Efficient Spectral-wise Multi-head Self-attention (E-SMSA). Unlike standard spatial attention, this mechanism treats each channel as a token. By applying average pooling to the "Key" and "Query" branches, the authors reduced the attention mechanism's complexity by 75% without sacrificing restoration quality. This modification makes TSF-Net twice as efficient as a vanilla transformer implementation, reducing inference time from 0.051s to 0.026s.

Experimental Results

The model was trained on the Div2K dataset and evaluated using 19 JVET common test sequences across various resolutions, from Class A1 (4K) to Class D.

  • Bitrate Savings: TSF-Net achieved an average BD-rate saving of 10.258% for the luma (Y) component under the All-Intra (AI) profile, significantly outperforming state-of-the-art models like DAM and Uformer.
  • Visual Quality: Qualitative analysis indicates that TSF-Net effectively reconstructs missing details suppressed during compression, such as reflections and fine textures, rather than merely blurring the noise.

Conclusion and Future Work

TSF-Net represents a significant step forward in learned in-loop filtering by effectively combining spatial and frequency features through an efficient transformer architecture. While currently focused on the Luma component and the All-Intra profile, future work will extend the model to Chroma components and temporal configurations like Random Access (RA) and Low Delay (LD). Additionally, the authors aim to explore scaling the transformer architecture for ultra-high resolutions where long-range correlations are even more critical.

Summary updated

Abstract

Block-based video codecs such as Versatile Video Coding (VVC)/H.266, High Efficiency Video Coding (HEVC)/H.265, Advanced Video Coding (AVC)/H.264 etc. inherently introduces compression artifacts. Although these codecs have in-loop filters to correct these distortions, they are not always effective due to the complexity of the noise. Recently, deep-learning approaches emerged as a promising solution for in-loop filtering. However, most of the previous approaches were designed solely for learning from images and neglected the high-frequency signals present in the reconstructed video frames. Furthermore, some previous methods employed a multi-level feature-extraction and feature-fusion strategy to enhance performance. However, they utilized complex feature-extractors while relying on naive feature-fusion methods. In this article, we propose a novel framework called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TSF-Net</i> , which jointly learns from both the pixel (spatial) and frequency-decomposed information and through powerful capability of a channel-wise transformer, it fuses both these information to improve performance. Our approach deviates from previous approaches by employing a simple feature-extractor coupled with an advanced transformer-based feature-fusion module. Simultaneously, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TSF-Net</i> introduces a few fundamental modifications in the multi-head self-attention module of the channel-wise transformer to make it computationally efficient. Our experimental results show that the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TSF-Net</i> achieves a Bjøntegaard Delta (BD) - bitrate saving of up to 10.258% for the luma (Y) component under all-intra (AI) profile outperforming the VVC baseline and other state-of-the-art methods. Moreover, the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TSF-Net</i> with an efficient channel-wise transformer is twice as efficient as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TSF-Net</i> with a vanilla channel-wise transformer.
BibTeX
@article{kathariya2023joint,
  author = {Birendra Kathariya and Zhu Li and Geert Van der Auwera},
  title = {{Joint Pixel and Frequency Feature Learning and Fusion via Channel-Wise Transformer for High-Efficiency Learned In-Loop Filter in VVC}},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2023},
  doi = {10.1109/tcsvt.2023.3323483},
}
Figure