From Rain Removal to Rain Generation

Hong Wang, Zongsheng Yue, Qi Xie, Qian Zhao, Deyu MengarXiv 2020 · OpenReview

Introduction

Single Image Rain Removal (SIRR) is a fundamental inverse problem in computer vision. To train effective deep learning models for this task, researchers require large-scale datasets of paired images: one containing rain and a corresponding clean "ground truth" background. However, capturing such pairs in the real world is nearly impossible, as environmental conditions cannot be perfectly controlled.

Consequently, the field has historically relied on synthetic datasets. These datasets often use simplistic mathematical models to overlay rain, failing to capture the complex variations in direction, thickness, and scale found in natural storms. This discrepancy creates a significant "domain gap," where models trained on synthetic data perform poorly when faced with real-world precipitation. This paper proposes a shift in perspective: instead of building more complex architectures to remove rain, we should first learn to generate more realistic rain.

The VRGNet Framework

The authors introduce the Variational Rain Generation Network (VRGNet), a Bayesian generative model designed to learn the underlying distribution of real rain. Unlike traditional methods that use fixed rules, VRGNet treats rain generation as a probabilistic process. It assumes a rainy image ($o$) is a combination of a rain layer ($r$) and a background ($b$).

The VRGNet architecture includes components for background restoration (BNet), rain inference (RNet), and rain generation (G).

The architecture consists of four primary components:

  1. RNet (Rain Inference): Estimates latent factors ($z$) from a rainy image.
  2. BNet (Background Restoration): Predicts the clean background ($b$).
  3. Generator ($G$): Renders a rain layer based on the inferred latent factors.
  4. Discriminator ($D$): An adversarial network that ensures generated rain is indistinguishable from real samples.

The model is trained using a variational lower bound (ELBO), ensuring that the generated rain follows the statistical distribution of the training data while maintaining physical plausibility.

Interpretable Rain Manifolds

A key innovation of VRGNet is its interpretable latent space. Unlike many "black box" generative models, specific dimensions of VRGNet’s latent space ($z$) correspond to physical traits such as rain direction, scale, and thickness. By navigating this "rain manifold," the generator can create diverse new rain patterns that were not present in the original training set but remain statistically accurate.

Visualizing how varying specific latent codes ($z_{22}$ and $z_{24}$) physically alters the scale and direction of the generated rain.

This interpretability allows for massive data augmentation. Moving a slider in the $z$-space physically tilts the rain or adjusts droplet thickness, proving the model has successfully learned the visual physics of rain appearance.

Key Results and Impact

The researchers evaluated VRGNet across several benchmark datasets, including synthetic sets like Rain100H and real-world data like SPA-Data. The results demonstrate a clear "data-centric" improvement:

  • Performance Gains: When existing deraining models (such as PReNet or DDN) were retrained using data augmented by VRGNet, their performance improved significantly. For instance, the DDN model’s PSNR on the Rain100L dataset rose from 32.38 dB to 35.56 dB.
  • Subjective Realism: In user studies, VRGNet’s synthetic rain was rated as more realistic than existing synthetic datasets. In some instances, it even outperformed the "real" SPA-Data, which can contain estimation artifacts.
  • Efficiency: The model effectively learns robust rain distributions even from limited real-world samples, reducing the burden of data collection.

Conclusion

VRGNet demonstrates that improving the quality and diversity of training data can be more effective than increasing the complexity of model architectures. While the current model uses a simplified additive approach ($o = r + b$), it provides a powerful foundation for future work. Subsequent iterations may incorporate more complex optical effects, such as the "veil effect" or splashing, by integrating VRGNet with physics-based rendering techniques.

Summary updated

Abstract

For the single image rain removal (SIRR) task, the performance of deep learning (DL)-based methods is mainly affected by the designed deraining models and training datasets. Most of current state-of-the-art focus on constructing powerful deep models to obtain better deraining results. In this paper, to further improve the deraining performance, we novelly attempt to handle the SIRR task from the perspective of training datasets by exploring a more efficient way to synthesize rainy images. Specifically, we build a full Bayesian generative model for rainy image where the rain layer is parameterized as a generator with the input as some latent variables representing the physical structural rain factors, e.g., direction, scale, and thickness. To solve this model, we employ the variational inference framework to approximate the expected statistical distribution of rainy image in a data-driven manner. With the learned generator, we can automatically and sufficiently generate diverse and non-repetitive training pairs so as to efficiently enrich and augment the existing benchmark datasets. User study qualitatively and quantitatively evaluates the realism of generated rainy images. Comprehensive experiments substantiate that the proposed model can faithfully extract the complex rain distribution that not only helps significantly improve the deraining performance of current deep single image derainers, but also largely loosens the requirement of large training sample pre-collection for the SIRR task.
BibTeX
@misc{wang2020from,
  author = {Hong Wang and Zongsheng Yue and Qi Xie and Qian Zhao and Deyu Meng},
  title = {{From Rain Removal to Rain Generation}},
  year = {2020},
  eprint = {2008.03580},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.03580},
}
Figure