Single Image Deraining With Continuous Rain Density Estimation

Lei Yu, Bishan Wang, Jingwei He, Gui-Song Xia, Wen Yang

TL;DR

This paper introduces CODE-Net, a novel deep learning architecture for single image deraining that tackles the challenges of non-uniform rain density and varying raindrop scales. By leveraging convolutional sparse coding and a continuous rain density estimation approach, CODE-Net achieves superior deraining performance and offers a more nuanced understanding of rain intensity compared to previous methods.

What Problem Does This Paper Address?

Capturing clear images in rainy weather is a persistent challenge for computer vision systems. Rain streaks can severely degrade image quality, leading to issues like over- or under-deraining, especially when the rain density is uneven across the image or when raindrops appear at different sizes. Existing methods often struggle with this variability, either by making simplifying assumptions about rain density (e.g., categorizing it into discrete levels like "light," "medium," or "heavy") or by failing to account for the diverse scales of raindrops. This paper aims to address these limitations by proposing a method that can handle continuous variations in rain density and multiple raindrop scales within a single, unified framework.

Methods

The core of the proposed solution lies in the CODE-Net (Continuous Density-guided Network). This network is built upon the principles of Convolutional Sparse Coding (CSC), a technique that represents image patches (in this case, rain streaks) as sparse linear combinations of learned "kernels" or dictionaries.

Here's a breakdown of the key methodological components:

  • Rain Streak Extraction and Denoising: CODE-Net first extracts potential rain streaks from the input image using convolutional layers. These extracted streaks are then denoised using a CSC-based approach.
  • Weighted Sparse Coding for Density Estimation: The crucial innovation is how rain density is incorporated. The paper posits that the sparsity of the coefficients obtained from CSC is directly related to rain density: lighter rain results in sparser coefficients. They adapt the standard iterative soft-thresholding algorithm (ISTA) for CSC by introducing learned weights for the sparse coefficients. These weights are generated using channel attention blocks, which learn to adaptively emphasize or de-emphasize coefficients based on their characteristics. This adaptive weighting implicitly considers the rain density.
  • Continuous Rain Density Estimation (RDE): Instead of discrete categories, CODE-Net estimates a continuous rain density value. This is achieved by averaging the learned weights from the channel attention blocks, providing a more granular measure of rain intensity. This RDE can also be used to evaluate deraining algorithms.
  • Multiscale Strategy (mCODE-Net): To handle varying raindrop scales, the authors extend CODE-Net to a multiscale version (mCODE-Net). This involves using multiple dictionaries with different kernel sizes to capture rain streaks at various scales, further improving the deraining performance.

Datasets

The paper utilizes several publicly available datasets for training and evaluation:

  • Training: 12,000 pairs of synthetic images and 1,800 pairs of real images from datasets like [8], [6], and a high-quality real rain dataset [7].
  • Testing:
    • Synthetic: Rain1200 [8], Rain12 [2], and Test1000 [7]. Rain1200 is highlighted as a particularly challenging dataset due to its mix of light, medium, and heavy rain.
  • Real-world: Various real-world rainy images are used for visual evaluation, as ground truth is unavailable.

No new benchmarks or datasets were introduced in this paper.

Key Insights and Ideas

  • Rain Density is Continuous, Not Discrete: A fundamental insight is that real-world rain intensity is a continuous spectrum, not a set of discrete levels. Previous methods that relied on discrete classifications (light, medium, heavy) inherently limited their adaptability. CODE-Net's continuous estimation offers a more accurate and flexible representation.
  • Sparse Coefficients as a Proxy for Rain Density: The paper establishes a strong link between the sparsity of coefficients in CSC and the density of rain. This insight is key to enabling density estimation without explicit labels.
  • Channel Attention for Adaptive Weighting: Using channel attention blocks to learn weights for sparse coefficients is an elegant way to implicitly incorporate rain density information into the deraining process. This allows the network to adapt its denoising strategy based on the estimated rain intensity.
  • Multiscale Approach for Diverse Raindrop Sizes: The extension to mCODE-Net effectively addresses the challenge of raindrops appearing at different scales, a factor often overlooked by single-scale deraining methods.

Key Results

  • Quantitative Performance: CODE-Net and mCODE-Net consistently outperform state-of-the-art deraining methods on synthetic datasets like Rain1200, Rain12, and Test1000, achieving higher PSNR and SSIM scores. For instance, on Rain1200, mCODE-Net achieved the best PSNR/SSIM (34.03/0.9281).
  • Qualitative Performance: Visual comparisons show that CODE-Net and mCODE-Net produce significantly better deraining results, effectively removing rain streaks while preserving background details and textures, especially in challenging heavy rain scenarios. They also avoid the over- or under-deraining issues common in other methods.
  • Continuous Density Estimation Validity: The RDE metric, derived from the learned weights, shows a clear correlation with rain intensity, decreasing significantly after deraining and providing a continuous measure that aligns with visual observations.
  • Efficiency: The paper demonstrates that CODE-Net can achieve competitive performance with a reasonable number of iterations (T=25), balancing accuracy and running time. mCODE-Net also achieves comparable or better results with fewer parameters than CODE-Net.

Limitations and Future Work

  • Failure Cases: The paper acknowledges that CODE-Net can sometimes be confused by rain-like structures, leading to overestimation of RDE and over-deraining. This is attributed to the global nature of the average pooling in the channel attention mechanism, which might miss local details.
  • Future Work:
    • Improved Weighting Block: The authors suggest improving the learning weight block by incorporating more local details, such as spatial structure and nonlocal self-similarity, to address the failure cases.
  • Extension to Other Tasks: The paper demonstrates the generality of CODE-Net by successfully applying it to image desnowing and exploring its use with perceptual loss for more realistic texture generation. This suggests potential for its application in other low-level vision tasks.
  • Precise Weights: Further research could focus on obtaining more precise weights for the sparse coefficients.

Summary updated

Abstract

Single image deraining (SIDR) often suffers from over/under deraining due to the nonuniformity of rain densities and the variety of raindrop scales. In this paper, we propose a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">co</i> ntinuous <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">de</i> nsity-guided network (CODE-Net) for SIDR. Particularly, it is composed of a rain streak extractor and a denoiser, where the convolutional sparse coding (CSC) is exploited to filter out noises from the extracted rain streaks. Inspired by the reweighted iterative soft-threshold (ISTA) for CSC, we address the problem of continuous rain density estimation by learning the weights with channel attention blocks from sparse codes. We further develop a multiscale strategy to depict rain streaks appearing at different scales. Experiments on synthetic and real-world data demonstrate the superiority of our methods over recent state-of-the-arts, in terms of both quantitative and qualitative results. Additionally, instead of quantizing rain density with several levels, our CODE-Net can provide continuous-valued estimations of rain densities, which is more desirable in real applications.
BibTeX
@article{yu2023single,
  author = {Lei Yu and Bishan Wang and Jingwei He and Gui-Song Xia and Wen Yang},
  title = {{Single Image Deraining With Continuous Rain Density Estimation}},
  journal = {IEEE Trans. Multimedia},
  year = {2023},
  url = {https://openreview.net/forum?id=PQgkmyuudn},
}
Figure