TL;DR
DGFusion introduces a novel depth-guided sensor fusion framework that dynamically adapts to spatially varying sensor reliability under adverse weather conditions. By framing multi-sensor segmentation as a multi-task problem with an auxiliary depth head, the network learns to leverage local depth cues to guide cross-modal attention at zero extra annotation or inference cost.
What Problem Does This Paper Address?
Robust perception is a cornerstone of safe autonomous driving. While modern self-driving cars use a suite of complementary sensors (cameras, lidars, radars, and event cameras), adverse weather like fog, rain, and snow degrades these sensors in unpredictable, spatially non-uniform ways. For instance, lidars provide highly accurate data nearby but fail rapidly at a distance in heavy fog or snow, while cameras struggle in poorly lit areas of a scene.
Current state-of-the-art sensor fusion methods typically treat sensor inputs uniformly across space or rely on global condition tokens (e.g., classifying an entire image as a "foggy night"). This overlooks the fact that sensor reliability changes from pixel to pixel based on the 3D structure of the scene. DGFusion addresses this gap by explicitly incorporating local depth context to dynamically adjust how much the network relies on each sensor across different regions of a scene.
Methods
DGFusion reformulates multimodal semantic/panoptic segmentation as a multi-task learning problem. The architecture consists of three main branches: semantic segmentation, condition representation, and an auxiliary depth estimation branch.
Key components include:
- Auxiliary Depth Head: During training, the network uses raw, sparse lidar data as ground truth to predict a dense depth map. Crucially, this depth head is discarded during inference, meaning it provides the benefits of geometric feature learning with zero extra computational cost at deployment.
- Depth-Guided Fusion Module: The network extracts intermediate depth features and pools them into local Depth Tokens (DT). Within local attention windows, these DTs are concatenated with RGB tokens and a global Condition Token (CT). This combined token set acts as the query mechanism for cross-attention, allowing the network to locally weigh the importance of secondary modalities (like radar or lidar) based on both the weather and the distance of the objects.
- Robust Depth Loss: Training on real-world adverse weather means the lidar ground truth itself is sparse and noisy. To combat this, the authors designed a specialized loss function combining:
- An outlier-robust $L_1$ loss with $\tau$-quantile filtering to ignore highly corrupted lidar returns.
- An RGB-edge-aware smoothness loss.
- A novel panoptic-edge-aware smoothness loss that prevents depth smoothing across distinct object instances of the same semantic class (e.g., two adjacent cars).
Datasets
The authors evaluated DGFusion on two challenging multi-sensor benchmarks:
- MUSES: A real-world dataset containing 2,500 multi-sensor scenes captured across eight distinct environmental conditions (combinations of rain, snow, fog, clear weather, and day/night) with high-quality human annotations for panoptic and semantic segmentation.
- DeLiVER: A synthetic dataset consisting of 7,885 scenes under adverse conditions and simulated sensor failures (like motion blur or lidar jitter), providing pixel-perfect depth maps.
Key Insights and Ideas
- Depth as a Proxy for Reliability: Instead of just treating lidar as a passive input, using it as a multi-task supervision target forces the network's internal layers to learn explicit geometric representations. Distance is the primary driver of sensor degradation in bad weather, so depth features naturally dictate sensor trustworthiness.
- Local + Global Synergy: A global weather token tells the network what the condition is (e.g., "it is snowing"), while the local depth token tells it where the condition matters most (e.g., "this object is far away, so ignore the lidar noise"). Together, they enable fine-grained, context-aware fusion.
- Panoptic-Aware Smoothing: Traditional semantic-guided depth smoothing only stops smoothing at class boundaries (e.g., road to car). By utilizing panoptic labels, DGFusion respects boundaries between individual instances of the same class, leading to much cleaner depth maps and sharper object localization.
Key Results
DGFusion achieved new state-of-the-art (SOTA) results across both benchmarks:
- MUSES Panoptic Segmentation: Achieved a Panoptic Quality (PQ) of 61.03%, outperforming the previous SOTA (CAFuser) by +1.33%. The performance gains were most prominent in adverse weather, showing a +2.57% PQ improvement in snow and +1.34% PQ in fog.
- MUSES Semantic Segmentation: Reached 79.5% mIoU, a +1.0% improvement over prior methods.
- DeLiVER: Outperformed existing fusion methods in both the camera-lidar-event (CLE) setup (51.6% mIoU) and the setup including dense depth maps (56.7% mIoU).
- Efficiency: At test time, DGFusion only adds 2.3% more parameters relative to its baseline and maintains a practical deployment speed of 6.83 FPS on an NVIDIA A6000 GPU.
Limitations and Future Work
While DGFusion demonstrates robust performance, the authors note a few limitations:
- Semantic Confusion: The network occasionally confuses highly similar semantic classes that share close geometric profiles, such as "wall" and "fence".
- Severe Occlusions: In low-visibility conditions (like heavy snowstorms), the model still struggles to detect heavily occluded objects, sometimes missing pedestrians hidden in the background.
For future work, extending this fine-grained local adaptivity directly to other long-range sensors like radar or incorporating temporal tracking context could further fortify perception in extreme environments.
Summary updated