Uncertainty Guided Multi-Scale Residual Learning-Using a Cycle Spinning CNN for Single Image De-Raining

Rajeev Yasarla, Vishal M. Patel

Abstract

Single image de-raining is an extremely challenging problem since the rainy image may contain rain streaks which may vary in size, direction and density. Previous approaches have attempted to address this problem by leveraging some prior information to remove rain streaks from a single image. One of the major limitations of these approaches is that they do not consider the location information of rain drops in the image. The proposed Uncertainty guided Multi-scale Residual Learning (UMRL) network attempts to address this issue by learning the rain content at different scales and using them to estimate the final de-rained output. In addition, we introduce a technique which guides the network to learn the network weights based on the confidence measure about the estimate. Furthermore, we introduce a new training and testing procedure based on the notion of cycle spinning to improve the final de-raining performance. Extensive experiments on synthetic and real datasets to demonstrate that the proposed method achieves significant improvements over the recent state-of-the-art methods.

Introduction

Single image de-raining is an "ill-posed" problem due to the high variability of rain streaks in density, direction, and size. Existing methods often struggle with location awareness, applying uniform processing that leads to over-smoothing in clear areas or residual streaks in heavy rain. Furthermore, deep architectures like U-Nets frequently introduce "ringing" or blocky artifacts near image edges. This paper introduces the Uncertainty guided Multi-scale Residual Learning (UMRL) network to address these challenges by combining multi-scale learning, uncertainty guidance, and cycle spinning.

Methods

The UMRL framework incorporates three primary innovations:

Multi-Scale Residual Learning

The network estimates the residual map (the rain streaks) at three resolutions (1.0, 0.5, and 0.25). This multi-scale approach allows the model to capture rain streaks of varying sizes more effectively than single-scale models.

Uncertainty Guidance

A core novelty of UMRL is the generation of per-pixel confidence maps at each scale. These maps provide a certainty score for the rain estimate. If the network is uncertain about a specific area, the confidence map directs subsequent layers to rely on other features, preventing error propagation and preserving fine textures.

Cycle Spinning

To eliminate boundary artifacts, the authors employ "cycle spinning," a classic signal processing technique. During testing, the image is cyclically shifted, de-rained, and then shifted back. Averaging multiple shifted results smooths out CNN-induced edge artifacts, significantly improving visual clarity.

Results and Discussion

The model was evaluated on standard synthetic benchmarks and real-world images. Quantitatively, UMRL achieved a PSNR of 29.77 dB on the Test-1 dataset, outperforming the previous state-of-the-art (DID-MDN) at 27.95 dB.

Visual comparison of de-raining results. (a, d) Input rainy images. (b, e) UMRL results without cycle spinning. (c, f) Final UMRL results with cycle spinning, showing improved edge clarity and detail recovery.

Qualitatively, UMRL excels at recovering sharp edges and complex textures, such as wooden walls or facial features, which are often blurred by other methods. The use of uncertainty guidance allows the network to be "self-correcting," preserving details that are not part of the rain streaks. While cycle spinning increases processing time (from 0.05s to 5.1s per image), it provides a substantial boost in visual quality.

Conclusion

The UMRL network demonstrates that combining uncertainty-guided learning with multi-scale residuals and classical signal processing techniques like cycle spinning can significantly advance single image de-raining. While the computational cost of cycle spinning is a limitation for real-time applications, the overall framework provides a robust foundation for image restoration tasks. Future work may focus on architectural optimizations to achieve cycle-spinning quality in a single forward pass.

Summary updated

Abstract

Single image de-raining is an extremely challenging problem since the rainy image may contain rain streaks which may vary in size, direction and density. Previous approaches have attempted to address this problem by leveraging some prior information to remove rain streaks from a single image. One of the major limitations of these approaches is that they do not consider the location information of rain drops in the image. The proposed Uncertainty guided Multi-scale Residual Learning (UMRL) network attempts to address this issue by learning the rain content at different scales and using them to estimate the final de-rained output. In addition, we introduce a technique which guides the network to learn the network weights based on the confidence measure about the estimate. Furthermore, we introduce a new training and testing procedure based on the notion of cycle spinning to improve the final de-raining performance. Extensive experiments on synthetic and real datasets to demonstrate that the proposed method achieves significant improvements over the recent state-of-the-art methods.
BibTeX
@inproceedings{yasarla2019uncertainty,
  author = {Rajeev Yasarla and Vishal M. Patel},
  title = {{Uncertainty Guided Multi-Scale Residual Learning-Using a Cycle Spinning CNN for Single Image De-Raining}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=dHnh3KZD3nRq},
}
Figure