Deep learning models for autonomous driving excel under clear skies. However, their performance degrades sharply when confronted with heavy rain, dense fog, or blinding snow. This vulnerability stems from a fundamental dataset imbalance: the vast majority of open-source autonomous driving data is collected under ideal weather conditions. As noted in The ApolloScape Open Dataset for Autonomous Driving and Its Application, this data scarcity severely limits our ability to train and evaluate robust perception models.
Driving a sensor-laden vehicle through a blizzard or torrential downpour is logistically challenging, hazardous, and prohibitively expensive. Furthermore, manually annotating 3D bounding boxes or semantic masks when visibility is near zero is incredibly difficult. To bridge this environmental diversity gap, researchers are turning to a mix of physics-informed sensor simulation, generative domain translation, and cross-weather knowledge transfer.
Physics-Informed Sensor Simulation
Instead of waiting for a storm, why not simulate one? The most direct way to generate adverse weather data is to model how environmental particles interact with sensor signals. This is particularly effective for active sensors like LiDAR, where the physics of light propagation is well-understood.
In Fog Simulation on Real LiDAR Point Clouds for 3D Object Detection in Adverse Weather, Christos Sakaridis et al. introduce a method to "foggify" clear-weather LiDAR point clouds. Instead of arbitrarily deleting points, the authors model the physical phenomena of signal attenuation (where fog absorbs or scatters the laser beam, reducing its returned energy) and backscattering (where the laser reflects off fog particles close to the sensor, creating phantom points). By applying these physical principles to existing clean datasets, they generate realistic foggy data without requiring manual field collection.
A similar approach is taken for winter conditions. In LiDAR Snowfall Simulation for Robust 3D Object Detection, Christos Sakaridis et al. simulate snowfall by accounting for both airborne flakes and the often-overlooked effect of wet, reflective ground surfaces. Similarly, Xun Huang et al. develop a physics-informed rain simulation method (DRET) in Sunshine to Rainstorm: Cross-Weather Knowledge Distillation for Robust 3D Object Detection. These methods work because they preserve the underlying 3D geometry of the scene while accurately degrading the sensor measurements exactly as nature would.
Generative Models for Structural and Density Control
While explicit physics equations work well for active sensors like LiDAR, modeling the visual complexity of rain streaks, puddles, and fog on camera images is incredibly complex. Here, generative adversarial networks (GANs) and diffusion models offer a data-driven alternative.
The primary challenge with using generative models for autonomous driving is maintaining structural consistency. If a generative model alters the position of a pedestrian or a vehicle while adding fog, the original ground-truth labels become useless. To solve this, Donghao Shi et al. propose DAB-GAN in Depth-aware unpaired image-to-image translation for autonomous driving test scenario generation using a dual-branch GAN. By incorporating depth information alongside RGB images, the dual-branch network ensures that the generated adverse weather effects respect the 3D layout of the scene, preventing structural distortion.
Other approaches leverage diffusion models or disentangled representations to control the severity of the weather. Unai Gurbindo et al. utilize the Instruct Pix2Pix diffusion model in Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pix to synthetically inject diverse weather effects into clear images via text prompts. For fine-grained control, Chi Zhang 0020 et al. present a self-supervised content-style disentanglement framework in Density-Aware Haze Image Synthesis by Self-Supervised Content-Style Disentanglement, allowing researchers to test perception models across a continuous spectrum of haze density.
Furthermore, generative techniques can operate entirely without real-world adverse weather examples during training. In Analogical Image Translation for Fog Generation, Rui Gong et al. use an analogical framework ($A : A' :: B : B'$) to learn the "gist" of fog generation from synthetic data and apply it directly to real-world clear images, achieving zero-shot translation. For LiDAR, Yang Wu et al. introduce WeatherGen in WeatherGen: A Unified Diverse Weather Generator for LiDAR Point Clouds via Spider Mamba Diffusion, utilizing a diffusion mechanism to generate diverse weather effects directly within 3D point clouds.
Knowledge Distillation and Domain Unification
Generating data is only half the battle; models must still learn to extract robust features from these degraded inputs. Two prominent strategies are cross-weather knowledge distillation and domain unification.
Knowledge distillation allows a model to inherit the clean-weather capabilities of a "teacher" network. In Sunshine to Rainstorm: Cross-Weather Knowledge Distillation for Robust 3D Object Detection, Xun Huang et al. introduce the SRKD framework. The intuition is simple: a teacher model trained on clear-weather data extracts high-fidelity, noise-free features. During training, a student model is fed the corresponding simulated rainy-weather data. By forcing the student's internal feature representations to match the teacher's clean features, the student learns to "see through" the rain and focus on the invariant geometry of the objects.
Alternatively, rather than trying to make an adverse-weather model behave like a clear-weather model, one can unify both domains into a shared space. In Real-to-Virtual Domain Unification for End-to-End Autonomous Driving, Yan Luo et al. propose DU-drive, a framework that translates real-world driving data into a simplified, virtual domain. By stripping away environmental noise and domain shifts before the perception step, the downstream driving model becomes inherently more robust to real-world environmental variations.
Real-World Benchmarks for Extreme Environments
While simulation and generation provide scalable training data, real-world datasets captured in extreme conditions remain the ultimate ground truth for validation. Recent dataset initiatives deliberately target the "fair-weather bias" by seeking out hazardous environments.
The Canadian Adverse Driving Conditions Dataset, introduced by Matthew Pitropov et al., provides thousands of frames of synchronized LiDAR and camera data explicitly captured during snowy Canadian winters, offering a rigorous benchmark for winter driving perception. To push localization algorithms beyond standard roads, Shibo Zhao et al. released the SubT-MRS Dataset: Pushing SLAM Towards All-weather Environments, which includes over 100km of trajectories through smoke-filled tunnels and snowy forests to test Simultaneous Localization and Mapping (SLAM) under zero-visibility constraints. Additionally, for airborne autonomy, Michael Fonder et al. developed Mid-Air: A Multi-Modal Dataset for Extremely Low Altitude Drone Flights, providing systematic variations in weather and season to ensure that low-altitude flight algorithms can generalize beyond flat, clear-weather assumptions.
Key Foundational Work
The ApolloScape Open Dataset for Autonomous Driving and Its Application (2019) — contributed 1 concept elements to this direction.
A*3D Dataset: Towards Autonomous Driving in Challenging Environments (2019) — contributed 1 concept elements to this direction.
Real-to-Virtual Domain Unification for End-to-End Autonomous Driving (2018) — contributed 1 concept elements to this direction.