Robust Multi-Modal Perception for Adverse Weather Conditions

679 concepts501 papers2 proposals

Autonomous vehicles (AVs) rely on a suite of sensors to "see" the world, but most current systems are "fair-weather friends." While high-resolution LiDAR and RGB cameras provide exceptional detail in clear conditions, they suffer from significant performance degradation in adverse weather like rain, fog, and snow. As noted by Kun Qian et al. in Robust Multimodal Vehicle Detection in Foggy Weather Using Complementary Lidar and Radar Signals, atmospheric particles cause signal scattering and attenuation, leading to noise that can blind a vehicle's perception system.

To achieve true all-weather autonomy, researchers are moving beyond simple sensor fusion toward architectures that can adapt to environmental degradation, synthesize realistic training data, and leverage weather-robust modalities like 4D radar.

The Physics of Sensor Failure

The degradation of perception in adverse weather is rooted in the physical interaction between sensor signals and the environment. LiDAR, which uses short-wavelength light to map 3D structures, is particularly vulnerable to "backscatter." As described by Matthew Pitropov et al. in the Canadian Adverse Driving Conditions Dataset, falling snow and fog reflect LiDAR pulses before they reach an object, creating a "wall" of noise points that the system may hallucinate as solid obstacles.

Cameras face a different set of challenges. Heavy rain introduces visual artifacts like streaks and spray, while fog reduces contrast and visibility range. Furthermore, physical obstructions like mud or dust on the lens—known as lens soiling—can completely obscure critical semantic information. Ganesh Sistu et al. argue in Let's Get Dirty: GAN Based Data Augmentation for Camera Lens Soiling Detection in Autonomous Driving that most perception models lack the "self-awareness" to identify when their input data has been corrupted by such contamination.

Synthesizing the Storm

A major bottleneck in developing robust models is the scarcity of labeled data captured in extreme weather. To bridge this gap, researchers use generative AI to "weatherize" clear-weather datasets. Unai Gurbindo et al. demonstrate in Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pix how diffusion models can synthetically generate realistic rain, fog, and snow effects.

Similarly, the authors of RainSD: Rain Style Diversification Module for Image Synthesis Enhancement using Feature-Level Style Distribution propose a module that combines mathematical modeling of rain streaks with style transfer. This allows models to train on diverse, safety-critical scenarios that are too dangerous or rare to capture frequently in the real world.

Leveraging Weather-Robust Modalities

When visual sensors fail, autonomous systems must rely on sensors that are physically immune to atmospheric interference. Radar, which uses longer wavelengths, can penetrate fog, rain, and smoke with ease.

Recent work has focused on 4D Radar, which provides not only range and heading but also elevation and instantaneous velocity. In RLNet: Adaptive Fusion of 4D Radar and Lidar for 3D Object Detection, Ruoyu Xu et al. show that adaptively fusing 4D radar with LiDAR allows the system to maintain high performance even when LiDAR signals are degraded.

Beyond just fusion, robust sensors can act as "teachers" for sensitive ones. Yujeong Chae et al. propose a framework in LiDAR-Based All-Weather 3D Object Detection via Prompting and Distilling 4D Radar where 4D radar is used to guide LiDAR models during training. This "knowledge distillation" enables a LiDAR-only model to learn features that are more resilient to weather-induced noise.

Adaptive Fusion and Resilience

The ultimate goal is a perception system that doesn't just combine sensors, but understands which sensor to trust at any given moment. Yang Lou et al. address this in Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving by encoding the uncertainty of each modality. If a camera is blinded by glare or LiDAR is cluttered by fog, the system dynamically weights the more reliable sensor.

Frameworks like SAMFusion by Edoardo Palladin et al. take this further by integrating gated Near-Infrared (NIR) cameras and radar using transformer-based adaptive blending. By treating sensor degradation not as a fatal error but as a variable to be managed, these architectures ensure that autonomous vehicles can navigate safely through the most challenging environmental conditions.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 478 papers
  2. Version 1 · 2026-10-04 · 618 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_environmental_sensor_degradation_a896547a,
  title = {environmental sensor degradation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/environmental-sensor-degradation/rdr_a896547aa58a4e15a297d7edbdddeb67},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_environmental_sensor_degradation_a896547a_v2,
  title = {environmental sensor degradation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/environmental-sensor-degradation/rdr_a896547aa58a4e15a297d7edbdddeb67/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake