DFMDA-Net: Dense Fusion and Multi-dimension Aggregation Network for Image Restoration

Huibin Yan, Shuoyao WangIJCAI 2024

Abstract

DFMDA-Net addresses two critical bottlenecks in U-shaped image restoration networks: inefficient feature fusion and limited multi-dimensional information aggregation. By introducing the Integration-Compression-Integration (ICI) mechanism and a Fourier-based Multi-dimension Aggregation (MDA) block, the proposed architecture achieves state-of-the-art performance across 16 benchmarking datasets for tasks including deraining, deblurring, and denoising.

Introduction

Image restoration is a fundamental yet ill-posed problem in computer vision, aimed at recovering a clean image from a degraded input (e.g., noisy, blurry, or rainy). While U-shaped encoder-decoder architectures have become the standard, they often struggle with suboptimal skip connections and limited receptive fields. Standard skip connections typically use simple addition or concatenation, leading to either information loss or excessive computational overhead. Furthermore, existing attention mechanisms often fail to efficiently aggregate global spatial and channel-wise information simultaneously.

Architecture and Methodology

The Dense Fusion and Multi-dimension Aggregation Network (DFMDA-Net) is designed to overcome these limitations through two novel components integrated into a hierarchical U-shaped framework.

Architecture of DFMDA-Net featuring ICI and MDA blocks.

Integration-Compression-Integration (ICI)

To improve feature fusion between the encoder and decoder, the ICI module replaces standard skip connections. It employs a "sandwich" structure: first integrating features to preserve information, then applying a compression layer for efficiency, and finally integrating again to maximize information utilization. This approach provides a superior trade-off between performance and parameter count compared to traditional concatenation.

Multi-dimension Aggregation (MDA)

The MDA block leverages the properties of the Fourier domain to achieve global information aggregation. By utilizing the Fast Fourier Transform (FFT), a single modification in the frequency domain influences the entire spatial domain, effectively providing an infinite receptive field. The MDA block uses group convolutions in the frequency domain to selectively enhance or suppress specific frequencies across channels, enabling simultaneous spatial and channel-wise feature aggregation without the quadratic complexity of standard self-attention.

Experimental Results

The authors evaluated DFMDA-Net across 16 benchmarking datasets covering image deraining, motion deblurring, defocus deblurring, and denoising.

Performance comparison across multiple restoration tasks.

Key findings include:

  • Deraining: DFMDA-Net achieved an average PSNR improvement of 0.27 dB over Restormer, with gains up to 1.27 dB on the Test100 dataset.
  • Deblurring: For motion deblurring, the model outperformed Restormer by 0.62 dB PSNR on average. In defocus deblurring, it achieved significant reductions in LPIPS scores compared to DeepRFT and SFNet.
  • Denoising: On the SIDD dataset for real-world noise, DFMDA-Net improved PSNR by 0.30 dB over previous state-of-the-art methods.

Conclusion

DFMDA-Net demonstrates that efficient "dense fusion" and global frequency-domain aggregation are powerful tools for image restoration. By moving beyond local spatial convolutions and simple skip connections, the model sets new performance benchmarks across a wide variety of tasks. Future work may involve optimizing the FFT-based blocks for variable input sizes and exploring the potential of DFMDA-Net as a unified foundation model for general image restoration.

Summary updated

BibTeX
@inproceedings{yan2024dfmda,
  author = {Huibin Yan and Shuoyao Wang},
  title = {{DFMDA-Net: Dense Fusion and Multi-dimension Aggregation Network for Image Restoration}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure