Multi-Scale Shape Adaptive Network for Raindrop Detection and Removal from a Single Image

Hao Luo, Qingbo Wu, King Ngi Ngan, Hanxiao Luo, Haoran Wei, Hongliang Li, Fanman Meng, Linfeng XuSensors 2020 · OpenReview

TL;DR

This paper introduces MSANet, a novel deep learning architecture for removing raindrops from single images. It addresses the limitations of existing methods by proposing a large-scale synthetic raindrop dataset (RaindropCityscapes) and a two-branch network that adaptively detects and removes diverse raindrops while preserving image details.

What Problem Does This Paper Address?

Images captured in rainy weather often suffer from poor visibility due to raindrops on camera lenses or windows. This degradation significantly impacts various computer vision tasks like object detection and person re-identification. While previous research has focused on rain streaks or mist, removing individual raindrops presents unique challenges due to their complex and varied shapes, scales, and transparency. Existing methods often struggle with these variations, leading to incomplete removal or damage to background details. Furthermore, publicly available raindrop datasets are limited in diversity and scale, hindering the development of robust models.

Methods

The authors propose a Multi-scale Shape Adaptive Network (MSANet), which consists of two main branches:

  • Raindrop Detection Branch: This branch aims to accurately estimate a pixel-level raindrop map. It incorporates a Receptive Field Block (RFB) to enhance the extraction of raindrop features, enabling better discrimination of raindrops with varying characteristics.
  • Raindrop Removal Branch: This branch focuses on removing the detected raindrops. It utilizes a Multi-scale Dilated Convolution Module (MDCM) to capture more texture details and semantic information, and a Multi-scale Densely Enhanced Deformable Module (DEDM) to adapt to complex raindrop shapes, scales, and transparency. The DEDM builds upon deformable convolutions, allowing for more flexible sampling of features.

The final derained image is obtained by fusing the original image and the output of the removal branch, guided by the raindrop map from the detection branch. This approach aims to preserve background details while effectively removing raindrops.

Datasets

  • RaindropCityscapes: A newly introduced large-scale synthetic dataset comprising 11,583 pairs of high-resolution raindrop and raindrop-free images. It is generated using an extended version of the ROLE algorithm and aims to cover a wide variety of raindrop characteristics (shape, scale, transparency, collision, fusion) and background scenarios, including different illumination conditions.
  • Cityscapes dataset: Used as the source for generating the raindrop-free images in RaindropCityscapes.
  • Real-world raindrop dataset: Collected from [21] for evaluating the method's performance on real-world scenarios.

Key Insights and Ideas

  • The importance of a diverse dataset: The authors highlight that existing raindrop datasets are insufficient to model the complexity of real-world raindrop scenarios, motivating the creation of RaindropCityscapes.
  • Adaptive shape handling is crucial: Simply applying shape-invariant filters is inadequate for diverse raindrops. The proposed MSANet's shape-adaptive nature, particularly through the DEDM, is key to its success.
  • Separating detection and removal: A two-branch approach allows for specialized processing: one branch focuses on accurately locating raindrops, while the other focuses on their removal, leading to better overall performance.
  • Deformable convolutions for flexibility: The use of deformable convolutions in the DEDM allows the network to adapt its sampling grid to the irregular shapes of raindrops, improving feature extraction.

Key Results

  • Quantitative Improvements: On the synthetic RaindropCityscapes dataset, MSANet achieved a PSNR of 40.45 dB and an SSIM of 0.9857, outperforming the previous state-of-the-art AGAN [21] by 2.13 dB and 4.9%. On a real-world dataset, MSANet achieved a PSNR of 25.32 and SSIM of 0.8270, also outperforming AGAN.
  • Qualitative Improvements: Visual comparisons show that MSANet effectively removes diverse raindrops, including small, large, dense, and fused ones, while preserving image details better than competing methods.
  • Extension to High-Level Tasks: When used as a preprocessing step for semantic segmentation (PSPNet) and object detection (Faster R-CNN) on the RaindropCityscapes dataset, MSANet-derained images led to significant improvements in mIoU, mAcc, mAP, and AP50 compared to other deraining methods.

Limitations and Future Work

  • Reflection Artifacts: The paper acknowledges that MSANet may struggle with preserving highly bright reflection artifacts in some rainy images.
  • Complex Raindrop Generation: The method might fail to remove certain "colorful rain-like reflection spots" due to limitations in existing training datasets that don't fully capture such complex raindrop conditions.
  • Future Work: The authors suggest collecting more raindrop samples with diverse reflection artifacts to address the identified limitations.

Summary updated

BibTeX
@inproceedings{luo2020multi,
  author = {Hao Luo and Qingbo Wu and King Ngi Ngan and Hanxiao Luo and Haoran Wei and Hongliang Li and Fanman Meng and Linfeng Xu},
  title = {{Multi-Scale Shape Adaptive Network for Raindrop Detection and Removal from a Single Image}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=zJvJcnnLPKj},
}
Figure