QFormer: An Efficient Quaternion Transformer for Image Denoising

Bo Jiang, Yao Lu, Guangming Lu, Bob Zhang

Abstract

Since Deep Convolutional Neural Networks (DCNNs) and Vision Transformer perform well in learning generalizable image priors from large-scale data, these models have been widely used in image denoising tasks. However, vanilla DCNNs and Transformer suffer from two problems. First, the vanilla DCNNs and Transformer only accumulate the output along the channel axis, ignoring the internal relationship among channels. This results in the severely inadequate color structure representation retrieved from color images. Secondly, the DCNNs or Transformer-based image denoising models usually have a large number of parameters, high computational complexity, and slow inference speed. To resolve these issues, this paper proposes a highly-efficient Quaternion Transformer (QFormer) for image denoising. Specifically, the proposed Quaternion Transformer Block (QTB) simplifies the typical Transformer from a multi-branch structure to an elaborately sequential structure mainly with quaternion transformations, to alternately capture both long-range dependencies and local contextual features with color structure information. Furthermore, the proposed QTB can also avoid considerable element-wise multiplications of computing the self-attention matrices. Thus, our QTB can significantly reduce the computational complexity and its sequential structure can further improve the practical inference speed. Comprehensive experiments demonstrate that the proposed QFormer produces state-of-the-art results in both denoising performance and efficiency.

Introduction

Modern image denoising architectures, primarily based on Deep Convolutional Neural Networks (DCNNs) and Transformers, often treat color images as stacks of independent monochromatic channels. This approach ignores the structural interrelationships between color channels, frequently leading to color distortion in the restored images. Additionally, high-performance Transformers are often computationally expensive due to self-attention mechanisms that scale quadratically with image resolution.

This paper introduces QFormer, a novel architecture designed to be both "color-aware" and "computationally lean" by integrating quaternion algebra into the Transformer framework.

Methodology

The core of the QFormer architecture is the Quaternion Transformer Block (QTB). The overall pipeline transforms color noisy images into the quaternion domain for processing, as illustrated in the following diagram.

The overall architecture of QFormer, showing the flow from color noisy images through the QFormer Network to produce denoised results.

Quaternion Representation and Processing

QFormer represents RGB pixels as "pure quaternions" ($I = Ri + Gj + Bk$). This mathematical framework allows the network to process all three color channels simultaneously, inherently preserving the inter-channel relationships.

The Quaternion Transformer Block (QTB)

The QTB replaces the standard multi-branch Transformer structure with a streamlined sequential design. It utilizes quaternion fully connected (Q-FC) layers to extract global features and quaternion convolutions (Q-Conv) to capture local context.

Detailed architecture of the Quaternion Transformer Block (QTB), highlighting the sequential extraction of global, local, and global-local features.

To maintain efficiency, QTB avoids the massive computational cost of standard self-attention matrices by using element-wise addition to aggregate global and local features. This captures long-range dependencies without the $O(N^2)$ bottleneck. Furthermore, the authors incorporate short skip connections to ensure "identity mapping," which is crucial for preserving high-frequency details that might otherwise be lost during quaternion transformations.

Experimental Evaluation

The proposed QFormer was evaluated on three major real-world noisy image benchmarks: the Smartphone Image Denoising Dataset (SIDD), PolyU, and Nam.

Performance and Efficiency

QFormer demonstrated superior performance compared to 14 state-of-the-art denoising methods:

  • Parameter Reduction: QFormer-T (Tiny) is approximately 22.8 times smaller than the Restormer model while achieving comparable results.
  • Denoising Accuracy: QFormer-B (Base) achieved a PSNR of 40.18 dB on the SIDD dataset, outperforming heavyweights like MIRNet.
  • Inference Speed: Ablation studies showed that QFormer's sequential structure significantly improves practical inference speed and reduces FLOPs compared to traditional UNet-based quaternion models.
  • Visual Quality: Qualitative results indicate that QFormer is particularly effective at preserving fine contours and edges, avoiding the blurring artifacts common in other models.

Conclusion

QFormer presents a highly efficient alternative to standard Transformer architectures for image denoising. By leveraging quaternion algebra, it achieves superior color structure representation and significant computational savings. Future research may explore extending this architecture to other complex domains, such as multispectral imagery or other image restoration tasks like super-resolution.

Summary updated

Abstract

Since Deep Convolutional Neural Networks (DCNNs) and Vision Transformer perform well in learning generalizable image priors from large-scale data, these models have been widely used in image denoising tasks. However, vanilla DCNNs and Transformer suffer from two problems. First, the vanilla DCNNs and Transformer only accumulate the output along the channel axis, ignoring the internal relationship among channels. This results in the severely inadequate color structure representation retrieved from color images. Secondly, the DCNNs or Transformer-based image denoising models usually have a large number of parameters, high computational complexity, and slow inference speed. To resolve these issues, this paper proposes a highly-efficient Quaternion Transformer (QFormer) for image denoising. Specifically, the proposed Quaternion Transformer Block (QTB) simplifies the typical Transformer from a multi-branch structure to an elaborately sequential structure mainly with quaternion transformations, to alternately capture both long-range dependencies and local contextual features with color structure information. Furthermore, the proposed QTB can also avoid considerable element-wise multiplications of computing the self-attention matrices. Thus, our QTB can significantly reduce the computational complexity and its sequential structure can further improve the practical inference speed. Comprehensive experiments demonstrate that the proposed QFormer produces state-of-the-art results in both denoising performance and efficiency. We hope that our work will encourage further research to explore the Quaternion Transformer architecture for image denoising tasks.
BibTeX
@inproceedings{jiang2024qformer,
  author = {Bo Jiang and Yao Lu and Guangming Lu and Bob Zhang},
  title = {{QFormer: An Efficient Quaternion Transformer for Image Denoising}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=7uQnGu9VAH},
}
Figure