From Snow to Rain: Evaluating Robustness, Calibration, and Complexity of Model-Based Robust Training

Josué Martínez-Martínez, Olivia Brown, Giselle Zeno, Pooya Khorrami, Rajmonda S. Caceres

Abstract: Robustness to natural corruptions remains a critical challenge for reliable deep learning, particularly in safety-sensitive domains. We study a family of model-based training approaches that leverage a learned nuisance variation model to generate realistic corruptions, as well as new hybrid strategies that combine random coverage with adversarial refinement in nuisance space. Using the Challenging Unreal and Real Environments for Traffic Sign Recognition dataset (CURE-TSR), with Snow and Rain corruptions, we evaluate accuracy, calibration, and training complexity across corruption severities. Our results show that model-based methods consistently outperform baselines Vanilla, Adversarial Training, and AugMix baselines, with model-based adversarial training providing the strongest robustness under across all corruptions but at the expense of higher computation and model-based data augmentation achieving comparable robustness with less computational complexity without incurring a statistically significant drop in performance. These findings highlight the importance of learned nuisance models for capturing natural variability, and suggest a promising path toward more resilient and calibrated models under challenging conditions.

Introduction

Deep neural networks (DNNs) are notoriously brittle when faced with environmental shifts. A small amount of snow on a traffic sign or rain on a camera lens can cause state-of-the-art models to fail, a significant barrier for safety-critical applications like autonomous driving. Beyond simple accuracy drops, these models are often overconfident in their mistakes—a failure of calibration. Traditional defenses like standard data augmentation or pixel-level adversarial training (AT) often fail to capture the complex, structured nature of real-world weather. This paper investigates whether learning a generative model of these "nuisances" (corruptions) can bridge the reliability gap.

Methods

The researchers evaluate three categories of training strategies:

  1. Analytical Baselines: Standard training (Vanilla), input-space Adversarial Training (AT), and AugMix (blending handcrafted filters).
  2. Model-Based Approaches: These utilize a generative model (MUNIT-style) to learn realistic corruptions from data.
  3. MDA (Model-based Data Augmentation): Applies a single random corruption from the generator.
  4. MRT (Model-based Robust Training): Samples $k$ corruptions and selects the most challenging one.
  5. MAT (Model-based Adversarial Training): Uses Projected Gradient Descent (PGD) to find worst-case corruption parameters in the generator's latent space.
  6. Hybrid Variants: Combinations such as MDAT and MRAT, which use random or "worst-of-k" sampling to initialize the adversarial search.

Experimental Setup

The study uses the CURE-TSR (Challenging Unreal and Real Environments for Traffic Sign Recognition) dataset, containing ~2.2 million images across 14 traffic sign classes. Evaluation focuses on Snow and Rain corruptions across five severity levels. Models are trained on "challenge-free" images augmented by the various methods and tested on corrupted test sets.

Results and Analysis

Robustness to Natural Corruptions

Model-based methods significantly outperform analytical baselines, particularly as corruption severity increases. Generative models capture the specific textures of snow and streaks of rain more effectively than handcrafted rules. For example, under severe snow (Severity 5), model-based methods maintain approximately 75% accuracy, while Vanilla and AT baselines drop to 55–65%.

Accuracy vs. Corruption Severity for Snow. Model-based methods (orange/pink) maintain significantly higher accuracy than baselines (blue) at high severity levels.

Calibration and Reliability

In addition to accuracy, model-based methods significantly improve the Expected Calibration Error (ECE). This ensures that the model's reported confidence better aligns with its actual accuracy. In challenging rain conditions, model-based adversarial training (MAT5) achieved an ECE of 20.6% at maximum severity, compared to over 30% for standard baselines.

ECE vs. Corruption Severity for Rain. Model-based methods show lower calibration error, indicating more reliable confidence estimates under severe weather conditions.

Efficiency vs. Performance

While MAT provides the strongest robustness, the authors identify MDA (random model-based augmentation) as a highly efficient alternative. Statistical tests (Welch’s t-test) revealed that in most cases, the performance difference between the computationally expensive MAT and the simpler MDA was not statistically significant. This suggests that the primary benefit stems from the realism of the learned nuisance model itself rather than the exhaustive adversarial search.

Discussion: The Severity Paradox

The research highlights a trade-off in training severity. Training on extreme corruptions (Severity 5) maximizes robustness but can degrade calibration on milder images. Conversely, training on moderate levels (Severity 2) produces better-calibrated models that are more aware of their uncertainty, though they are less resilient to extreme conditions.

Conclusion

Model-based training strategies offer a promising path toward resilient and calibrated deep learning models. By leveraging learned nuisance models, these approaches outperform traditional augmentation and adversarial training in both accuracy and reliability. Future work will explore extending these methods to more complex architectures like Vision Transformers and a broader range of environmental conditions.

Summary updated

Abstract

Robustness to natural corruptions remains a critical challenge for reliable deep learning, particularly in safety-sensitive domains. We study a family of model-based training approaches that leverage a learned nuisance variation model to generate realistic corruptions, as well as new hybrid strategies that combine random coverage with adversarial refinement in nuisance space. Using the Challenging Unreal and Real Environments for Traffic Sign Recognition dataset (CURE-TSR), with Snow and Rain corruptions, we evaluate accuracy, calibration, and training complexity across corruption severities. Our results show that model-based methods consistently outperform baselines Vanilla, Adversarial Training, and AugMix baselines, with model-based adversarial training providing the strongest robustness under across all corruptions but at the expense of higher computation and model-based data augmentation achieving comparable robustness with $T$ less computational complexity without incurring a statistically significant drop in performance. These findings highlight the importance of learned nuisance models for capturing natural variability, and suggest a promising path toward more resilient and calibrated models under challenging conditions.
BibTeX
@misc{martinezmartinez2026from,
  author = {Josué Martínez-Martínez and Olivia Brown and Giselle Zeno and Pooya Khorrami and Rajmonda S. Caceres},
  title = {{From Snow to Rain: Evaluating Robustness, Calibration, and Complexity of Model-Based Robust Training}},
  year = {2026},
  eprint = {2601.09153},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.09153},
}
Figure