Adaptive Frequency Enhancement Network for Single Image Deraining

Fei Yan, Yuhong He, Keyu Chen, En Cheng, Jikang Ma

TL;DR

This paper introduces AFENet, a novel deep learning network for single image deraining that tackles the challenge of uneven rain degradation across different frequency components of an image. By adaptively enhancing features across low, mid, and high frequencies, AFENet achieves superior rain removal and detail reconstruction compared to existing methods.

What Problem Does This Paper Address?

Images captured in rainy conditions suffer from various degradations like rain streaks, raindrops, and rain accumulation. While many deep learning methods have improved image deraining in the spatial domain, they often struggle with the fact that rain degradation affects different frequency components of an image unevenly. For instance, rain streaks are typically high-frequency, while rain accumulation might be lower-frequency. Existing methods, by focusing primarily on spatial information, fail to adequately address these frequency-specific challenges, leading to residual rain or distorted details in complex rainy scenes. This paper aims to bridge this gap by proposing a deraining network that explicitly considers and adaptively enhances features across the entire frequency spectrum.

Methods

The core of the proposed solution is the Adaptive Frequency Enhancement Network (AFENet), which comprises three key modules:

  • Frequency Decomposition Module (FDM): This module adaptively decomposes the input rainy image into three distinct frequency bands: low, middle, and high. Unlike traditional methods like DCT or wavelet transforms, AFENet uses convolutions with varying receptive fields to achieve this decomposition. It employs an implicit downsampling strategy using convolutional layers with different strides to extract these frequency components, aiming for efficiency and better representation learning.
  • Frequency Enhancement Module (FEM): Designed to boost features within each frequency band, the FEM utilizes a novel transformer block called the Multi-Dconv Head Transposed Attention (MDTA). MDTA leverages self-attention mechanisms across channels and integrates depth-wise convolutions to capture both global and local image features effectively. This module is crucial for enhancing the representation capabilities of each frequency branch.
  • Frequency Aggregation Module (FAM): This module is responsible for interactively fusing the enhanced features from the different frequency branches. It uses a multi-scale feature generation unit and incorporates Pyramid Attention to refine details, especially in the higher-frequency components. The FAM ensures a comprehensive and integrated enhancement across all frequency bands by allowing selective fusion and progressively leveraging lower-frequency representations to enhance higher ones.

Datasets

The paper uses several standard datasets for training and evaluation:

  • Training: Rain13K dataset (a collection of 13,712 clean-rain image pairs from various synthetic datasets).
  • Testing:
    • Synthetic: Rain100H, Rain100L, Test100, Test1200, Test2800.
  • Real-world: Real15 and Real300.
  • Benchmark: RainDS dataset (comprising 3600 pairs with rain streaks, raindrops, and both).

Key Insights and Ideas

  • Frequency Domain Matters: The paper strongly emphasizes that rain degradation is not uniform across all frequencies. Simply enhancing spatial features isn't enough; a frequency-aware approach is critical for effective deraining.
  • Adaptive Enhancement is Key: Instead of a one-size-fits-all approach, AFENet adaptively enhances features in each frequency band, recognizing that different frequencies require different treatments.
  • Transformer Power for Features: The integration of transformer-based attention mechanisms (MDTA) within the FEM allows for more effective capture of global and local image context, which is vital for complex deraining tasks.
  • Progressive Enhancement: The strategy of using lower-frequency enhanced features to help restore higher-frequency details is an elegant way to tackle the inherent difficulty of recovering fine-grained textures.

Key Results

  • State-of-the-Art Performance: AFENet achieves the highest performance on multiple benchmarks, outperforming 24 existing state-of-the-art deraining methods.
  • Quantitative Gains: On the Rain13k benchmark, AFENet surpasses LDRCNet by 0.23 dB in PSNR and 0.007 in SSIM. For the RainDS benchmark, it shows significant improvements over DGUNet, with gains of 2.08 dB in PSNR for RS scenes and 1.11 dB for RD scenes.
  • Real-World Efficacy: The method also demonstrates superior performance on real-world datasets (Real15 and Real300) using no-reference metrics like NIQE and BRISQUE.
  • Qualitative Superiority: Visual comparisons show AFENet effectively removes residual rain streaks and excels at restoring image details, producing visually appealing results.

Limitations and Future Work

  • Ablation Study Insights: The ablation studies highlight the crucial role of each component: frequency decomposition, interaction between branches, and feature aggregation. Removing any of these significantly degrades performance, underscoring the integrated nature of the proposed approach.
  • Future Directions: While not explicitly detailed as future work, the paper's focus on adaptive frequency enhancement and the use of transformer modules suggest potential for extending this framework to other image restoration tasks that suffer from frequency-dependent degradations. The success of the MDTA module also points towards further exploration of transformer architectures for feature extraction in image processing.

Summary updated

Abstract

Image deraining aims to improve the visibility of images damaged by rainy conditions, targeting the removal of degradation elements such as rain streaks, raindrops, and rain accumulation. While numerous single image deraining methods have shown promising results in image enhancement within the spatial domain, real-world rain degradation often causes uneven damage across an image's entire frequency spectrum, posing challenges for these methods in enhancing different frequency components. In this paper, we introduce a novel end-to-end Adaptive Frequency Enhancement Network (AFENet) specifically for single image deraining that adaptively enhances images across various frequencies. We employ convolutions of different scales to adaptively decompose image frequency bands, introduce a feature enhancement module to boost the features of different frequency components and present a novel interaction module for interchanging and merging information from various frequency branches. Simultaneously, we propose a feature aggregation module that efficiently and adaptively fuses features from different frequency bands, facilitating enhancements across the entire frequency spectrum. This approach empowers the deraining network to eliminate diverse and complex rainy patterns and to reconstruct image details accurately. Extensive experiments on both real and synthetic scenes demonstrate that our method not only achieves visually appealing enhancement results but also surpasses existing methods in performance. The source code is available at https://github.com/yanfefei/AFENet.
BibTeX
@misc{yan2024adaptive,
  author = {Fei Yan and Yuhong He and Keyu Chen and En Cheng and Jikang Ma},
  title = {{Adaptive Frequency Enhancement Network for Single Image Deraining}},
  year = {2024},
  eprint = {2407.14292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.14292},
}
Figure