Scale-Space Feature Recalibration Network for Single Image Deraining

Pengpeng Li, Jiyu Jin, Guiyue Jin, Lei FanSensors 2022 · DOI

TL;DR

This paper introduces SFR-Net, a novel deep learning architecture for single image deraining that excels at preserving image details. By integrating multi-scale feature extraction with a new Subspace Coordinated Attention Mechanism (SCAM), SFR-Net effectively recalibrates rain streak features, leading to superior deraining performance and enhanced visual quality compared to existing methods.

What Problem Does This Paper Address?

Accurate and reliable visual information is crucial for applications like intelligent security and autonomous driving. However, rainy weather significantly degrades image quality, introducing blur and obscuring scene details. While many single image deraining algorithms exist, they often struggle to retain fine image details, leading to a trade-off between rain removal and image fidelity. This paper aims to address this limitation by developing a network that can effectively remove rain streaks while preserving intricate background information.

Methods

The core of the proposed solution is the Scale-space Feature Recalibration Network (SFR-Net). Key components include:

  • Multi-scale Extraction Recalibration Block (MERB): This block is designed to capture rich multi-scale rain streak features. It utilizes dilated convolutions with varying kernel sizes to extract features at different scales.
  • Subspace Coordinated Attention Mechanism (SCAM): This novel attention mechanism is embedded within MERB. It combines a Coordinated Attention Recalibration Mechanism (CARM) and a Subspace Attention Mechanism (SAM). CARM captures cross-channel and spatial feature information, while SAM further refines these features by dividing them into subspaces. This process aims to recalibrate rain streak features and eliminate redundant information, thereby enhancing the transfer of important features.
  • Dense Connectivity and Cross-Layer Feature Fusion: The overall SFR-Net architecture employs dense connections and cross-layer feature fusion. This allows for repeated utilization of feature maps, improving the network's understanding of the image and mitigating gradient vanishing issues.

Datasets

The paper utilizes several established datasets for training and evaluation:

  • Synthetic Datasets:
    • Rain100L: 200 training pairs, 100 test pairs (light rain).
  • Rain100H: 1800 training pairs, 100 test pairs (heavy rain, 5 streak types).
  • Rain800: 700 training pairs, 100 test pairs.
  • Rain1400: 12,600 training pairs, 1400 test pairs (14 streak types).
  • Real-World Datasets:
    • Li et al. (Scene 1): 185 real images.
  • Li et al. (Scene 2): 34 real images.

No new benchmark datasets were introduced in this paper.

Key Insights and Ideas

  • The Importance of Multi-Scale Feature Interaction: The paper highlights that rain streaks exhibit self-similarity across different scales. Effectively capturing and fusing these multi-scale features is crucial for accurate deraining.
  • Attention for Feature Refinement: The SCAM mechanism demonstrates that intelligently recalibrating features, rather than just extracting them, can significantly improve deraining performance. By focusing on relevant features and discarding noise, the network can better preserve background details.
  • Subspace Attention for Focused Feature Learning: Dividing features into subspaces within the attention mechanism allows for more targeted learning and recalibration, preventing information loss and improving the network's ability to distinguish between rain streaks and background content.
  • Dense Connections for Robust Feature Propagation: The use of dense connections throughout the network ensures that features are effectively reused and propagated, leading to a more robust understanding of the image and preventing degradation of information as it passes through deeper layers.

Key Results

The proposed SFR-Net achieved state-of-the-art performance on both synthetic and real-world datasets.

  • Quantitative Results: On synthetic datasets like Rain100L and Rain100H, SFR-Net consistently outperformed existing methods in terms of PSNR and SSIM. For instance, on Rain100L, the proposed method achieved a PSNR of 39.67/0.988, surpassing other methods. On Rain100H, it achieved 30.90/0.912.
  • Qualitative Results: Visual comparisons showed that SFR-Net effectively removes a vast majority of rain streaks while preserving image details and avoiding blurring or color distortion, which were observed in other methods. The network produced more natural and enjoyable recovered images, especially on complex real-world rainy scenes.

Limitations and Future Work

  • Limitations:
    • The paper primarily focuses on rain streaks. While the proposed method shows robustness across various rain densities and types, it does not explicitly address other severe weather conditions like haze or snow.
  • The ablation study on SCAM subspaces suggests that while increasing subspaces generally improves performance, the gains become marginal after a certain point (n=8), indicating a potential trade-off between complexity and performance.
  • Future Work:
    • The authors suggest upgrading the design to handle more complex weather scenarios, including haze and snow, aiming for a more generalized performance.
  • Future work will explore inter-domain adaptation techniques to achieve domain migration and weight assignment using multi-source synthetic datasets, further enhancing robustness and generalization.

Summary updated

Abstract

Computer vision technology is increasingly being used in areas such as intelligent security and autonomous driving. Users need accurate and reliable visual information, but the images obtained under severe weather conditions are often disturbed by rainy weather, causing image scenes to look blurry. Many current single image deraining algorithms achieve good performance but have limitations in retaining detailed image information. In this paper, we design a Scale-space Feature Recalibration Network (SFR-Net) for single image deraining. The proposed network improves the image feature extraction and characterization capability of a Multi-scale Extraction Recalibration Block (MERB) using dilated convolution with different convolution kernel sizes, which results in rich multi-scale rain streaks features. In addition, we develop a Subspace Coordinated Attention Mechanism (SCAM) and embed it into MERB, which combines coordinated attention recalibration and a subspace attention mechanism to recalibrate the rain streaks feature information learned from the feature extraction phase and eliminate redundant feature information to enhance the transfer of important feature information. Meanwhile, the overall SFR-Net structure uses dense connection and cross-layer feature fusion to repeatedly utilize the feature maps, thus enhancing the understanding of the network and avoiding gradient disappearance. Through extensive experiments on synthetic and real datasets, the proposed method outperforms the recent state-of-the-art deraining algorithms in terms of both the rain removal effect and the preservation of image detail information.
BibTeX
@inproceedings{li2022scale,
  author = {Pengpeng Li and Jiyu Jin and Guiyue Jin and Lei Fan},
  title = {{Scale-Space Feature Recalibration Network for Single Image Deraining}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22186823},
}
Figure