Deep Learning for Seeing Through Window With Raindrops

Yuhui Quan, Shijie Deng, Yixin Chen, Hui Ji

Abstract

Removing raindrops from images taken through glass is a challenging yet critical task for outdoor vision systems. This paper presents a novel Convolutional Neural Network (CNN) that leverages a Joint Physical Shape + Channel Attention (JPCA) module to restore image clarity. By incorporating geometric priors—specifically the roundness and closedness of raindrops—the model effectively identifies and removes distortions while preserving background details. Experimental results on benchmark datasets demonstrate that this approach outperforms existing state-of-the-art methods, including GAN-based architectures, in both quantitative metrics and visual quality.

Introduction

Adherent raindrops on windows or camera lenses pose a significant challenge for computer vision applications, particularly in autonomous driving and surveillance. These droplets act as complex lenses, refracting and reflecting light, which leads to severe visual distortions or complete occlusion of the background scene.

Examples of images captured through a window with raindrops, illustrating various levels of distortion and occlusion across different environments.

Restoring a clean image from a single corrupted frame is a "highly ill-posed" problem. Traditional methods often fail to generalize across varying raindrop sizes and shapes, frequently leaving behind artifacts or failing to detect transparent droplets. This paper addresses these issues by introducing a shape-driven attention mechanism that utilizes the physical properties of raindrops to guide the restoration process.

Methodology: Joint Physical Shape + Channel Attention

The proposed architecture is an encoder-decoder CNN enhanced by the Joint Physical Shape + Channel Attention (JPCA) module. This module integrates two distinct types of attention to improve feature extraction and image reconstruction.

1. Shape-Driven (Physical) Attention

Unlike standard attention mechanisms that learn weights purely from data, the shape-driven module incorporates a "torque operator" to search for geometric features characteristic of raindrops: convexness and contour closedness. Since raindrops are typically elliptical or round, these physical priors allow the network to accurately locate and mask raindrop regions. This is achieved through specialized convolutional kernels designed to learn elliptical parameters.

2. Channel Re-calibration (Channel Attention)

To complement spatial attention, the model employs a channel re-calibration module (similar to Squeeze-and-Excitation blocks). This component assigns weights to different feature channels, enabling the network to prioritize background textures over raindrop-related artifacts.

3. Robust Edge Detection

To support the shape-driven module, the authors implemented a custom edge detector. This detector is more resilient to environmental noise than standard gradient-based methods, ensuring reliable droplet boundary detection even in low-contrast or poorly lit scenes.

Experimental Setup and Datasets

The model was rigorously tested using the following:

  • Qian et al. Dataset: A benchmark consisting of 1,110 image pairs (raindrop-corrupted vs. clean ground truth), created by photographing scenes through glass before and after applying water spray.
  • Real-World Testing: The model was also evaluated on "wild" images captured via mobile phones in actual rainy conditions to verify its generalization capabilities beyond controlled environments.

Results and Discussion

The proposed JPCA-based model was compared against several leading methods, including Eigen et al.’s CNN and the GAN-based AttentGAN.

Quantitative Performance

On the Qian et al. benchmark, the proposed model achieved a Peak Signal-to-Noise Ratio (PSNR) of 30.86 dB and a Structural Similarity Index (SSIM) of 0.9263. These results surpass AttentGAN, which recorded 30.55 dB and 0.9023, respectively. Ablation studies confirmed the importance of the JPCA module; its removal led to a nearly 1 dB drop in PSNR.

Qualitative Comparison

Visual analysis shows that the proposed method produces significantly cleaner images with fewer "ghosting" artifacts or blurry patches than previous state-of-the-art techniques.

Extracted figure from page 2 of Deep Learning for Seeing Through Window With Raindrops

Interestingly, the authors found that their targeted CNN approach outperformed Generative Adversarial Networks (GANs). While GANs are known for producing visually "realistic" textures, they often introduce hallucinated artifacts in the context of raindrop removal, whereas the proposed CNN maintains higher fidelity to the actual background.

Conclusion and Future Work

This research demonstrates that integrating physical priors into deep learning architectures can significantly enhance performance for specific image restoration tasks. By combining shape-driven and channel attention, the model effectively "sees through" raindrops.

Future work will focus on:

  • Handling Opaque Occlusions: Improving the reconstruction of areas completely blocked by opaque reflections.
  • Computational Efficiency: Optimizing multi-scale kernels for real-time deployment on automotive hardware.
  • Temporal Consistency: Extending the framework to video sequences to ensure stable, flicker-free restoration in dynamic environments.

Summary updated

Abstract

When taking pictures through glass window in rainy day, the images are comprised and corrupted by the raindrops adhered to glass surfaces. It is a challenging problem to remove the effect of raindrops from an image. The key task is how to accurately and robustly identify the raindrop regions in an image. This paper develops a convolutional neural network (CNN) for removing the effect of raindrops from an image. In the proposed CNN, we introduce a double attention mechanism that concurrently guides the CNN using shape-driven attention and channel re-calibration. The shape-driven attention exploits physical shape priors of raindrops, i.e. convexness and contour closedness, to accurately locate raindrops, and the channel re-calibration improves the robustness when processing raindrops with varying appearances. The experimental results show that the proposed CNN outperforms the state-of-the-art approaches in terms of both quantitative metrics and visual quality.
BibTeX
@inproceedings{quan2019deep,
  author = {Yuhui Quan and Shijie Deng and Yixin Chen and Hui Ji},
  title = {{Deep Learning for Seeing Through Window With Raindrops}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00255},
}
Figure