ERF: A Benchmark Dataset for Robust Semantic Segmentation Under Extreme Rainfall Conditions

Xin Yang, Xin Zhang, Xinchao WangAAAI 2025 · DOI

TL;DR

This paper introduces the Extreme RainFall (ERF) dataset, the first benchmark specifically designed for robust semantic segmentation under violent rainfall conditions. They demonstrate that current state-of-the-art models, even those leveraging foundation models, struggle significantly under these extreme conditions, highlighting a critical gap in autonomous driving perception systems.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurate perception, particularly semantic segmentation, to understand their surroundings. As climate change intensifies, extreme weather events like violent rainfall are becoming more frequent. These events severely degrade visual input through phenomena like water flow, veiling effects, and raindrop distortions, making it difficult for current computer vision models to reliably identify objects. While existing datasets cover light to heavy rain, there's a significant lack of data and evaluation for the most extreme rainfall conditions, leaving a critical safety gap for autonomous systems operating in vulnerable regions.

Methods

The authors propose the Extreme RainFall (ERF) dataset for both image and video semantic segmentation. They then benchmark existing models across four categories:

  1. Image-based Foundation Models: Leveraging large pre-trained models like CLIP, SAM, EVA02, and DINOv2 as feature extractors.
  2. Image-based Domain Generalization (DG) Methods: Models trained to generalize to unseen domains without target-specific fine-tuning (e.g., HRDA, HGFormer, PASTA, Rein).
  3. Image-based Domain Adaptation (DA) Methods: Models that explicitly learn to bridge the gap between source (ideal weather) and target (rainy) domains (e.g., DAFormer, HRDA, MIC, CDAC).
  4. Video-based Methods: Models that leverage temporal information across frames to improve robustness (e.g., DA-VSN, TPS, SFC).

They evaluate these methods on their new ERF dataset using the mean Intersection over Union (mIoU) metric.

Datasets

  • Training/Evaluation: The primary contribution is the Extreme RainFall (ERF) dataset. It contains 14,757 unlabeled frames and 100 labeled frames captured during four distinct violent rainfall events in Singapore. The dataset is annotated with 18 classes, following the Cityscapes protocol (excluding 'train').
  • Comparison/Training: Existing datasets mentioned for comparison or as potential sources for training include:
  • Cityscapes (Cordts et al. 2016) - for general urban scene understanding and as a source for DG/DA methods.
  • ACDC (Sakaridis et al. 2021) - covers various adverse conditions, including rain.
  • BDD100K (Yu et al. 2020) - a large-scale driving dataset.
  • RainCityscapes (Hu et al. 2019) - synthetic rain effects.
  • Raincouver (Tung et al. 2017) - rainy conditions.
  • nuScenes (Caesar et al. 2020) - autonomous driving.
  • RaidaR (Jin et al. 2021) - real-world rainy images.
  • MVSS (Ji et al. 2023) - video dataset with various conditions.
  • Rainy WCity (Zhong et al. 2022) - heavy rainy conditions.
  • Viper (Richter et al. 2017) - ideal weather video dataset, used as a source for video-based methods.

Key Insights and Ideas

  • Violent Rain is a Unique Challenge: The paper clearly illustrates that violent rainfall causes degradations (water flow, severe veiling) that are qualitatively different and more severe than those seen in heavy rain, making existing models struggle.
  • Foundation Models Show Promise: Models that leverage large foundation models (especially DINOv2) generally perform better than other image-based methods, suggesting that broad pre-training is beneficial for robustness.
  • Domain Generalization Outperforms Domain Adaptation (in this extreme case): Surprisingly, DG methods (like Rein) integrated with foundation models outperformed DA methods. This suggests that the domain gap in violent rain is so large that traditional DA struggles, while DG's focus on invariant features is more effective.
  • Temporal Information Isn't Enough (Yet): Video-based methods, despite using temporal cues, showed a significant drop in performance compared to image-based methods, indicating that current video approaches are not yet robust enough for this extreme scenario.
  • Unlabeled Data Helps: Comparing HRDA in DG vs. DA settings, the DA version (which used unlabeled target data) showed improvement, hinting at the potential of leveraging unlabeled violent rain data for adaptation.

Key Results

  • No Model is Robust: Across all tested categories, no model achieved satisfactory performance on the ERF dataset.
  • Foundation Models Lead: DINOv2, when used as a foundation model, achieved the highest mIoU scores among the tested image-based methods.
  • Rein Excels in DG: The Rein method, a DG approach integrated with DINOv2, outperformed other DG methods and even some DA methods.
  • Video Methods Lag: Video-based methods exhibited significantly lower mIoU scores compared to image-based approaches. For instance, DA-VSN achieved a maximum mIoU of 36.5% on the ERF dataset, while DINOv2 reached 59.8%.
  • Qualitative Failures: The paper shows qualitative examples where models misclassify objects (e.g., car as 'fence' or 'road') due to severe water flow and veiling effects, even for top-performing models.

Limitations and Future Work

  • Dataset Size: While the ERF dataset is the first of its kind for violent rain, it has a relatively small number of labeled frames (100) compared to some other large-scale datasets.
  • Model Performance: The paper explicitly states that none of the tested models are capable of withstanding the extreme challenges of violent rainfall, indicating a significant performance gap.
  • Future Directions:
    • Developing new models specifically designed for violent rainfall conditions.
  • Exploring foundation model-based domain adaptation methods to better bridge the large domain gap.
  • Improving video-based methods by enhancing their backbones and better incorporating temporal information for dynamic rainfall degradation.
  • Investigating how to effectively utilize unlabeled target data for adaptation in extreme conditions.

Summary updated

BibTeX
@inproceedings{yang2025erf,
  author = {Xin Yang and Xin Zhang and Xinchao Wang},
  title = {{ERF: A Benchmark Dataset for Robust Semantic Segmentation Under Extreme Rainfall Conditions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.33007},
}
Figure