TL;DR
This paper introduces QuDeC, a novel deep learning approach for removing rain streaks from single images. QuDeC uniquely incorporates a "quality-label map" to understand the spatial variation of rain distortion, guided by confidence measures, leading to significant improvements over existing methods.
What Problem Does This Paper Address?
Removing rain streaks from images is a critical task for many computer vision applications, such as autonomous driving and surveillance, as rain significantly degrades image quality. While existing methods tackle this problem, they often struggle with the diverse nature of rain streaks (varying size, direction, and density) across different image regions. A key limitation is their failure to explicitly consider the location-specific impact of rain, leading to artifacts like under- or over-deraining. This paper aims to address this by developing a method that accounts for the spatial variability of rain distortion.
Methods
The core of the proposed method, QuDeC (Quality-based single image Deraining using Confidence measure), builds upon a previous work (UMRL) by introducing two key innovations:
- Location-Quality-Label Map: Instead of just estimating the rain streaks (residual map), QuDeC also estimates a "location-quality-label map." This map assigns a distortion level (low, medium, high, represented by green, blue, and red) to different patches of the image, effectively capturing how rain affects various regions. This is achieved by using the Natural Image Quality Evaluator (NIQE) score as a proxy for distortion.
- Confidence Measures: To make the process robust, QuDeC computes confidence scores for both the estimated residual maps and the location-quality-label maps. These confidence scores act as a gating mechanism, down-weighting erroneous estimations and allowing the network to learn more effectively. This enables "soft labeling" of distortion levels, making the method more resilient to inaccuracies in the quality labels.
The QuDeC network architecture consists of an encoder and two decoders. Decoder D1 estimates the residual rain streaks and their confidence maps, while Decoder D2 estimates the location-quality-label maps and their confidence scores. These outputs are then combined to produce the final de-rained image. The loss function is designed to jointly optimize for accurate residual estimation and correct distortion level labeling, incorporating confidence scores to guide the learning process.
Datasets
The paper utilizes several standard datasets for training and evaluation:
- Synthetic Datasets:
- DIDMDN dataset ([1]): Contains 12,000 images with low, medium, and high rain levels.
- Dataset from [23]: 700 training images.
- Dataset from [3]: 1800 rainy images for training.
- Real-World Datasets: 100 real-world rainy images from [1], [23], and [3] were used for qualitative evaluation.
The authors also describe a pseudo-code for generating ground-truth location-quality-label maps using NIQE scores on patches from the DIDMDN training dataset. They also mention training a "Generation-of-Labels Network (GLN)" to automatically generate these maps.
Key Insights and Ideas
- Spatial Distortion Matters: The most significant insight is that rain streaks are not uniform across an image. Different regions experience varying levels of distortion, and explicitly modeling this spatial variation is crucial for effective de-raining.
- Quality as a Prior: Using a "quality-label map" derived from image quality metrics (like NIQE) as a prior for de-raining is a novel way to guide the network. This map provides contextual information about the rain's impact.
- Confidence is Key: The introduction of confidence measures for both residual and quality estimations is vital. It allows the network to dynamically adjust its learning based on the reliability of its intermediate predictions, leading to more robust results.
- Joint Optimization: Formulating de-raining as a joint task of estimating residual rain and distortion levels, rather than separate tasks, allows for a more holistic approach and better integration of information.
Key Results
- Quantitative Performance: QuDeC consistently outperforms state-of-the-art methods on synthetic datasets, achieving higher average PSNR and SSIM scores. For instance, it outperforms UMRL+cycle-spinning [16] by an average of 0.3dB.
- Qualitative Performance: The method produces visually superior results on both synthetic and real-world images. It effectively removes rain streaks without introducing artifacts, preserving image details and textures, unlike some competing methods that exhibit over- or under-deraining. Figures 11 and 12 showcase these improvements.
- Ablation Study: An ablation study (Table II and Figure 13) demonstrates the contribution of each component. Adding the Decoder D2 (for distortion level estimation) improves performance by approximately 0.37dB, and adding the LCN (Label Confidence Network) further boosts it by 0.2dB.
Limitations and Future Work
- NIQE Limitations: The paper acknowledges that NIQE scores, while useful, might not be perfectly tailored for rain streak distortion specifically. The thresholds used for generating labels can also vary between datasets.
- GLN for Label Generation: The authors propose training a Generation-of-Labels Network (GLN) to automatically generate location-quality-label maps, suggesting a path towards more automated and robust label generation.
- Generalizability: While tested on various datasets, further exploration of QuDeC's performance on extremely diverse and challenging rain conditions could be a future direction.
Summary updated