TL;DR
LiDSeg is a modular framework that enhances real-time semantic segmentation in adverse weather by using an auxiliary reconstruction task during training to "guide" the network. By discarding the heavy reconstruction decoder at inference time, it achieves state-of-the-art robustness against fog, rain, and snow with minimal computational overhead, making it ideal for edge deployment in autonomous systems.
What Problem Does This Paper Address?
Semantic segmentation is a cornerstone of autonomous driving, but its performance typically craters in adverse weather. Rain, fog, and snow introduce artifacts that corrupt the features models rely on for scene understanding.
Existing solutions generally fall into two camps:
- Restore-then-Segment: These pipelines explicitly dehaze or derain an image before segmenting it. This adds significant latency and often introduces "over-smoothed" artifacts that actually hurt segmentation accuracy.
- Domain Adaptation: These methods try to align feature distributions between clear and "messy" weather. However, they are often computationally expensive and struggle with representation bottlenecks.
The authors of LiDSeg sought a "third way": a method that learns weather-robust features without the latency of full-image restoration at runtime.
Methods
LiDSeg introduces a dual-branch architecture consisting of a Segmentation Branch (using standard lightweight models like BiSeNetV2 or DDRNet) and a Guidance Branch.
The core innovation lies in its asymmetric training strategy:
- Training Phase: The Guidance Branch uses a linear-attention encoder and an auxiliary reconstruction decoder. The model is trained on paired images (e.g., a foggy image and its clear ground truth). The reconstruction task forces the guidance encoder to learn how to "see through" the weather.
- Inference Phase: The reconstruction decoder is thrown away. Only the lightweight guidance encoder remains, providing robust features to the main segmentation branch.
To merge these two branches, the authors developed the Lightweight Gated Fusion (LGF) module. Instead of simple concatenation, LGF uses a channel-wise gating mechanism (inspired by Squeeze-and-Excitation) to dynamically decide how much guidance information is needed at different scales.
Datasets
The models were trained and evaluated on several key benchmarks:
- Foggy Cityscapes: Used for synthetic pretraining with perfectly aligned clear/foggy pairs.
- ACDC (Adverse Conditions Dataset with Correspondences): A challenging real-world dataset covering fog, night, rain, and snow.
- Foggy Zurich & Foggy Driving: Used to test "zero-shot" generalization—evaluating the model on real-world fog without having seen those specific datasets during training.
Key Insights and Ideas
- Reconstruction as Regularization: The authors treat image reconstruction not as an end goal, but as a powerful regularizer. By forcing the encoder to be capable of reconstruction, it inherently learns features that are stable across different weather domains.
- Linear Attention for the Edge: To keep the guidance branch efficient, they replaced standard $O(N^2)$ self-attention with a linear-attention formulation ($O(N)$), which is much better suited for high-resolution images on resource-constrained hardware.
- Channel Coherence: The authors observed that weather degradations like fog and rain are largely "channel-coherent." This insight allowed them to use lightweight channel-wise gating in the LGF module rather than expensive spatial attention, saving significant FLOPs.
- Feature Stability: Quantitative analysis showed that LiDSeg significantly reduces "feature drift"—the internal representation of a scene stays much more consistent whether it's clear or foggy compared to baseline models.
Key Results
LiDSeg achieves a remarkable balance between accuracy, robustness, and speed:
- Robustness: On Foggy Cityscapes, LiDSeg limited mIoU degradation to just 2.5–2.8 percentage points, reducing the performance drop of baseline models by more than 5×.
- Performance: LiDSeg-PIDNet achieved 75.07% mIoU on the ACDC test server, matching the performance of the much larger HRNet-W48 while being significantly more efficient.
- Efficiency: The framework is "plug-and-play," adding only 1.9–3.1M parameters. On an NVIDIA Jetson AGX Orin, the DDRNet variant runs at over 200 FPS (at 512x1024 resolution).
- Generalization: In zero-shot tests on Foggy Zurich, LiDSeg outperformed prior unsupervised domain adaptation (UDA) methods, despite those methods having seen the target domain during training.
Limitations and Future Work
The primary limitation identified is nighttime performance. Unlike fog or rain, nighttime degradation is dominated by low illumination and non-uniform artificial lighting, which breaks the "channel-coherence" assumption. While LiDSeg still improves over baselines at night, the gains are less dramatic than in other conditions.
Future work directions include:
- Developing illumination-aware gating to better handle nighttime scenes.
- Exploring self-supervised versions of the guidance branch to remove the need for synthetic paired training data.
- Extending the framework to video segmentation to leverage temporal consistency.
Summary updated