DDT: Dual-branch Deformable Transformer for Image Denoising

Kangliang Liu, Xiangcheng Du, Sijie Liu, Yingbin Zheng, Xingjiao Wu, Cheng Jin

Abstract

Image denoising remains a fundamental challenge in low-level computer vision, requiring a delicate balance between preserving fine structural details and capturing global context. While Convolutional Neural Networks (CNNs) and Vision Transformers have advanced the state-of-the-art, they often suffer from limited receptive fields or prohibitive quadratic computational complexity. This paper introduces the Dual-branch Deformable Transformer (DDT), an efficient architecture designed to capture both local and global dependencies in parallel. By utilizing a novel deformable attention mechanism, DDT achieves linear computational complexity $O(HW)$ relative to image size, significantly reducing the FLOPs and parameter count compared to heavyweights like Restormer and IPT while maintaining competitive performance.

Introduction

Image denoising is the process of recovering a clean image from a noisy observation. Historically, CNNs have been the standard for this task; however, their reliance on local convolution kernels limits their ability to model long-range dependencies. Transformers have emerged as a powerful alternative due to their self-attention mechanism, which facilitates global modeling.

Despite their success, standard Transformers face a "quadratic complexity" bottleneck: as image resolution increases, the computational cost grows quadratically. Previous attempts to mitigate this—such as window-based attention or channel-wise attention—often sacrifice either global context or spatial locality. Furthermore, dense self-attention is often redundant in images where many pixels share similar information. DDT addresses these challenges by combining a dual-branch processing strategy with adaptive sampling via deformable attention.

Methodology

The DDT Architecture

DDT utilizes a hierarchical U-Net-like encoder-decoder structure. The core of the network is the DDT Block (DDTB), which departs from the traditional sequential processing of local and global features. Instead, it employs a parallel dual-branch structure to process information simultaneously.

Dual-branch Structure

To efficiently capture multi-scale information, the feature map is split into two parallel paths:

  1. Local Branch: This branch divides the feature map into non-overlapping patches and performs self-attention within each patch. This focuses the model on high-frequency details and fine textures.
  2. Global Branch: To capture long-range dependencies without the cost of full-image attention, this branch samples pixels from identical relative positions across different patches. This provides a "big picture" view of the image structure.

Deformable Attention (DA)

To further enhance efficiency and reduce redundancy, DDT incorporates Deformable Attention. Rather than attending to every pixel in a fixed grid, the model employs a "Position Sub-network" to learn spatial offsets. These offsets allow the model to adaptively sample only the most informative points in the feature map. This sparse sampling strategy significantly reduces the number of key-value pairs, enabling the model to achieve linear complexity $O(HW)$.

Depth-wise Feed-Forward Network (DFFN)

Standard Transformer blocks use a Feed-Forward Network (FFN) consisting of two linear layers. DDT enhances this by introducing depth-wise convolutions between the linear layers. This addition strengthens the model's ability to handle local spatial structures and improves the overall stability of the restoration process.

Experimental Evaluation

Datasets and Implementation

The model was evaluated across several standard benchmarks:

  • Real-World Noise: The Smartphone Image Denoising Dataset (SIDD) and the Darmstadt Noise Dataset (DND).
  • Synthetic Noise: Gaussian noise was added to the CBSD68, Kodak24, McMaster, and Urban100 datasets at varying noise levels ($\sigma=15, 25, 50$).

Performance and Efficiency

DDT demonstrates a superior balance between restoration quality and computational overhead:

  • Computational Efficiency: On the SIDD dataset, DDT achieved a PSNR of 39.83 dB using only 86G FLOPs. In comparison, Restormer achieved 40.02 dB but required nearly double the computation at 155G FLOPs.
  • Model Size: DDT is significantly more lightweight than its peers, utilizing 18.4M parameters. This is a substantial reduction compared to Uformer-B (50.9M) and the massive IPT (115.3M).
  • Synthetic Benchmarks: On the Urban100 dataset, DDT outperformed the IPT model while using approximately 6x fewer parameters and 4.4x fewer FLOPs, highlighting its effectiveness in handling complex structural patterns.

Ablation Studies

Ablation experiments confirmed the necessity of the dual-branch approach. The researchers found that the parallel combination of local and global branches improved PSNR by approximately 0.05-0.06 dB compared to using either branch in isolation. This suggests that local and global contexts are complementary and most effective when processed in tandem.

Conclusion

The Dual-branch Deformable Transformer (DDT) represents a significant step toward efficient high-resolution image restoration. By replacing dense, quadratic self-attention with a parallel dual-branch structure and adaptive deformable sampling, DDT achieves state-of-the-art efficiency. While it sits slightly behind the absolute highest-performing models in raw PSNR on specific benchmarks, its drastically lower computational footprint makes it a highly practical solution for real-world applications. Future research may explore scaling this architecture for 4K/8K resolution tasks or extending the dual-branch deformable mechanism to other restoration domains such as deblurring and super-resolution.

Summary updated

BibTeX
@inproceedings{liu2023ddt,
  author = {Kangliang Liu and Xiangcheng Du and Sijie Liu and Yingbin Zheng and Xingjiao Wu and Cheng Jin},
  title = {{DDT: Dual-branch Deformable Transformer for Image Denoising}},
  booktitle = {ICME},
  year = {2023},
  doi = {10.1109/icme55011.2023.00470},
}
Figure