Adherent Raindrop Detection and Removal in Video

Shaodi You, Robby T. Tan, Rei Kawakami, Katsushi IkeuchiCVPR 2013 · OpenReview

Abstract

Raindrops adhered to a windscreen or window glass can significantly degrade the visibility of a scene. Detecting and removing raindrops will, therefore, benefit many computer vision applications, particularly outdoor surveillance systems and intelligent vehicle systems. In this paper, a method that automatically detects and removes adherent raindrops is introduced. The core idea is to exploit the local spatio-temporal derivatives of raindrops. First, it detects raindrops based on the motion and the intensity temporal derivatives of the input video. Second, relying on an analysis that some areas of a raindrop completely occludes the scene, yet the remaining areas occludes only partially, the method removes the two types of areas separately. For partially occluding areas, it restores them by retrieving as much as possible information of the scene, namely, by solving a blending function on the detected partially occluding areas using the temporal intensity change. For completely occluding areas, it recovers them by using a video completion technique. Experimental results using various real videos show the effectiveness of the proposed method.

Introduction

Raindrops on a camera lens are a major challenge for outdoor computer vision systems, such as those used in autonomous vehicles or surveillance. Unlike falling rain, "adherent" raindrops remain stationary on the glass, occluding and deforming the background scene. These drops are notoriously difficult to handle due to their shape variability, transparency (which distorts the background), and focus issues, as their proximity to the lens often results in significant blurring.

Adherent raindrops on a glass surface significantly degrade visibility.

Methodology

The proposed approach consists of two primary phases: detection and removal.

Detection: The "Fish-eye" and "Low-pass" Clues

The researchers identified two key spatio-temporal signatures of raindrops:

  1. Motion Contraction: A clear raindrop acts as a miniature fish-eye lens, compressing a large portion of the environment into a small area. Consequently, the motion observed inside a raindrop is 20 to 30 times slower than the motion in the rest of the frame.
  2. Temporal Smoothing: Blurred raindrops act as a mixture of light rays from across the scene, effectively functioning as a temporal low-pass filter. This means the intensity of a pixel covered by a blurred drop changes much more slowly over time than a clear background pixel.

The algorithm utilizes SIFT flow to track motion and calculates temporal intensity derivatives to identify these signatures. Level sets and geometric constraints (e.g., roundness and size) are then applied to finalize the raindrop masks.

Automated detection of raindrops using the proposed spatio-temporal signatures.

Removal: Blending and Completion

Once detected, the system treats the raindrop area based on the severity of the occlusion:

  • Partial Occlusion: For areas where the background is still partially visible, the authors solve a blending function in the frequency domain (using Discrete Cosine Transform) to "peel back" the raindrop layer and restore the background.
  • Full Occlusion & Glare: For areas where the scene is totally blocked or washed out by glare, the system employs a video completion (inpainting) technique to fill in the gaps using information from adjacent frames.

Experimental Results

The method was evaluated using controlled lab data and real-world footage from hand-held, vehicle-mounted, and static surveillance cameras.

  • Detection Accuracy: The proposed method significantly outperformed previous techniques (e.g., Roser et al. and Kurihata et al.), particularly in handling various shapes and highly blurred drops.
  • Efficiency: The removal process is notably faster than general-purpose video completion. The proposed method takes approximately 5 seconds per frame, compared to 2 minutes per frame for the Wexler et al. method, while producing fewer visual artifacts.
  • Visual Quality: Qualitative results demonstrate a dramatic improvement in visibility, which is essential for downstream tasks like feature tracking or stereo correspondence.

Conclusion

This paper presents a robust method for adherent raindrop removal by exploiting the unique optical properties of raindrops. By distinguishing between partial and full occlusion, the system achieves high-quality restoration with significant computational efficiency. Future work will address highly dynamic raindrops where the static assumption may not hold.

Summary updated

Abstract

Raindrops adhered to a windscreen or window glass can significantly degrade the visibility of a scene. Detecting and removing raindrops will, therefore, benefit many computer vision applications, particularly outdoor surveillance systems and intelligent vehicle systems. In this paper, a method that automatically detects and removes adherent raindrops is introduced. The core idea is to exploit the local spatio-temporal derivatives of raindrops. First, it detects raindrops based on the motion and the intensity temporal derivatives of the input video. Second, relying on an analysis that some areas of a raindrop completely occludes the scene, yet the remaining areas occludes only partially, the method removes the two types of areas separately. For partially occluding areas, it restores them by retrieving as much as possible information of the scene, namely, by solving a blending function on the detected partially occluding areas using the temporal intensity change. For completely occluding areas, it recovers them by using a video completion technique. Experimental results using various real videos show the effectiveness of the proposed method.
BibTeX
@inproceedings{you2013adherent,
  author = {Shaodi You and Robby T. Tan and Rei Kawakami and Katsushi Ikeuchi},
  title = {{Adherent Raindrop Detection and Removal in Video}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2013},
  url = {https://openreview.net/forum?id=r1Eko1fOZB},
}
Figure