Hierarchical-Level Rain Image Generative Model Based on GAN

Zhenyuan Liu, Tong Jia, Xingyu Xing, Jianfeng Wu, Junyi Chen

TL;DR

This paper introduces RCCycleGAN, a novel generative model that creates realistic images of varying rain intensities (light, medium, heavy) for training and testing autonomous vehicle perception systems. By introducing rain intensity labels and optimizing training strategies, RCCycleGAN significantly outperforms existing methods in generating high-fidelity rain images, crucial for improving the safety of self-driving cars in adverse weather.

What Problem Does This Paper Address?

Autonomous vehicles rely heavily on visual perception systems to navigate safely. However, these systems struggle in adverse weather conditions like rain, which can degrade image quality and lead to performance limitations, posing a significant risk to the Safety Of The Intended Functionality (SOTIF). To address this, researchers need high-quality, diverse datasets of rainy conditions to train and validate these perception systems. Existing datasets are often limited, either lacking multi-level rain intensity, being synthetically generated with low authenticity, or having limited diversity. This gap makes it challenging to thoroughly test and improve the robustness of autonomous driving perception systems against various rain scenarios.

Methods

The core of this work is the Rain Conditional CycleGAN (RCCycleGAN), a hierarchical-level rain image generative model built upon the Generative Adversarial Network (GAN) framework, specifically extending CycleGAN. The key innovations include:

  • Hierarchical Rain Intensity Control: Unlike standard GANs, RCCycleGAN incorporates rain intensity labels (light, medium, heavy) as conditions for generation. This allows for controllable and specific rain simulation.
  • Rain Mask Identification Network (RMI): To better capture rain-specific features, an unsupervised RMI network is introduced. This network, based on LSTM and residual modules, helps extract rain masks from both natural and simulated images, aiding the generator in learning rain patterns more effectively.
  • Optimized Generator and Discriminator: The generator (using a U-Net structure) and discriminator networks are designed to handle the conditional input (image + rain intensity label + rain mask). The paper adopts the Least Squares GAN (LSGAN) loss function for improved training stability and faster convergence compared to traditional GANs.
  • Cycle Consistency Loss: Similar to CycleGAN, a cycle consistency loss is maintained to ensure that translating an image from sunny to rainy and back to sunny (or vice-versa) results in an image close to the original.
  • Image Feature Recognition Network: A pre-trained VGG16 network is used to extract features from both real and simulated images, serving as an additional constraint to ensure structural similarity.
  • Training Strategy Enhancements: Based on the DerainCycleGAN baseline, three modifications were made: introducing rain intensity labels, replacing the sigmoid activation function in the discriminator to weaken its performance, and implementing a suppression strategy to balance the adversarial training between the generator and discriminator.

Datasets

The authors introduce a new dataset called the Natural Multilevel Rain Dataset (NMRD).

  • NMRD: This dataset consists of natural rain images collected under various weather conditions, specifically sunny, light rain, medium rain, and heavy rain. It includes images with diverse objects like vehicles, pedestrians, and bicycles. The dataset is made publicly available.
  • Comparison Datasets: The paper also references and compares against existing datasets like KITTI, Nuscense, BDD100K, Cityscapes, CADC, Boreas, SPA, SIRR, Rain800, and Rain12600, highlighting their limitations in terms of rain intensity levels or authenticity.

Key Insights and Ideas

  • Controllable Rain Generation is Crucial: The ability to generate images with specific, hierarchical rain intensities is vital for targeted testing of perception systems, going beyond simple "rainy" or "not rainy" classifications.
  • Rain Masks as Key Information: Explicitly modeling and extracting rain masks significantly aids GANs in learning the complex patterns of rain streaks and occlusions, leading to more realistic results.
  • Balancing the Adversarial Game: The performance of GANs is highly sensitive to the balance between the generator and discriminator. Weakening the discriminator and adjusting training strategies are effective ways to prevent mode collapse and achieve better overall generation quality.
  • Natural Data is Superior: The NMRD dataset, composed of real-world rain images, demonstrates the advantage of using authentic data for training and evaluation, leading to models that produce more realistic outputs.

Key Results

  • Quantitative Improvements: RCCycleGAN achieved significant improvements over baseline models:
  • Compared to CycleGAN, PSNR increased by 2.58 dB and SSIM by 18%.
  • Compared to DerainCycleGAN, PSNR increased by 0.74 dB and SSIM by 8%.
  • Qualitative Improvements: RCCycleGAN generates rain images that better retain structural information, exhibit less distortion and noise, preserve more color information, and capture finer local details (shadows, reflections) compared to CycleGAN and DerainCycleGAN, making them closer to real-world scenarios.
  • Ablation Study Findings: The ablation experiments confirmed the effectiveness of the model tuning. Introducing rain intensity labels initially decreased generator performance, but subsequent optimizations (discriminator weakening, suppression strategy) led to overall model improvement, highlighting the importance of a balanced adversarial setup.

Limitations and Future Work

  • Limitations:
    • The paper primarily assesses performance based on image quality metrics (PSNR, SSIM).
  • The generalization capability of the model to diverse public datasets is not extensively explored.
  • Future Work:
    • Object Detection Evaluation: The authors plan to evaluate the generated images by incorporating object detection results to assess the model's performance in a more practical autonomous driving context.
  • Generalization Testing: Training and testing RCCycleGAN on existing public datasets will be pursued to verify its generalization capabilities across different data distributions.

Summary updated

Abstract

<div class="section abstract"><div class="htmlview paragraph">Visual perception systems for autonomous vehicles are exposed to a wide variety of complex weather conditions, among which rainfall is one of the weather conditions with high exposure. Therefore, it is necessary to construct a model that can efficiently generate a large number of images with different rainfall intensities to help test the visual perception system under rainfall conditions. However, the existing datasets either do not contain multilevel rainfall or are synthetic images. It is difficult to support the construction of the model. In this paper, the natural rainfall images of different rainfall intensities were first collected and produced a natural multilevel rain dataset. The dataset includes no rain and three levels (light, medium and heavy) of rainfall with the number of 629, 210, 248 and 193 respectively, totaling 1280 images. The dataset is open source and available online via: <a href="https://github.com/raydison/natural-multilevel-rain-dataset-NMRD" target="_blank">https://github.com/raydison/natural-multilevel-rain-dataset-NMRD</a>. Subsequently, a hierarchical-level rain image generative model, rain conditional CycleGAN (RCCycleGAN), is constructed. RCCycleGAN is based on the generative adversarial network (GAN), which can generate images of light, medium and heavy rain by inputting no rain images into the model. In the process of model tuning, a total of three modifications are made based on the DerainCycleGAN, including introducing different rainfall intensity labels, replacing the activation function, and adjusting the training strategy. Compared with the two baseline models, CycleGAN and DerainCycleGAN, the peak signal-to-noise ratio (PSNR) of RCCycleGAN on the test dataset is improved by 2.58 dB and 0.74 dB, and the structural similarity (SSIM) is improved by 18% and 8%, respectively. The ablation experiments are also carried out and validate the effectiveness of the model tuning.</div></div>
BibTeX
@inproceedings{liu2024hierarchical,
  author = {Zhenyuan Liu and Tong Jia and Xingyu Xing and Jianfeng Wu and Junyi Chen},
  title = {{Hierarchical-Level Rain Image Generative Model Based on GAN}},
  booktitle = {SAE},
  year = {2024},
  doi = {10.4271/2024-01-7034},
}
Figure