TL;DR
LiDAR semantic segmentation models achieve impressive accuracy on clean datasets but degrade significantly when deployed in unpredictable real-world environments. This paper introduces the first large-scale robustness benchmarks—SemanticKITTI-C and SemanticPOSS-C—to evaluate 11 state-of-the-art models across 16 real-world corruptions. Leveraging insights from this comprehensive study, the authors design RLSeg, a robust voxel-based model that significantly improves out-of-distribution generalizability without sacrificing clean-data performance.
What Problem Does This Paper Address?
Autonomous driving systems rely heavily on LiDAR semantic segmentation to perceive their surroundings and make safety-critical decisions. While current deep learning models excel on standard, pristine validation datasets, real-world deployment exposes them to severe data corruptions. These include adverse weather (fog, snow), sensor measurement noise, and cross-device discrepancies (e.g., running a model trained on a high-resolution 64-beam LiDAR on a vehicle equipped with a cheaper 16-beam sensor).
Prior robustness research in 3D vision has primarily focused on synthetic, single-object datasets like ModelNet40, which fail to capture the complexity of large-scale driving scenes. This paper fills the gap by establishing a rigorous framework to analyze how different input representations, network architectures, and training strategies withstand real-world environmental and hardware variations.
Methods
The authors systematically benchmark 11 existing LiDAR semantic segmentation models, categorizing them by their primary input representations:
- Projection-based methods: Project 3D point clouds onto 2D planes (Range View or Bird's-Eye View) to leverage efficient 2D CNNs.
- Point-based methods: Directly process raw, irregular point clouds using localized geometric aggregation functions.
- Voxel-based methods: Convert point clouds into 3D voxel grids and apply sparse 3D convolutions.
Based on a comprehensive analysis of these architectures, the authors introduce RLSeg (Robust LiDAR Segmentation). RLSeg utilizes a single-representation voxel backbone (MinkowskiNet) combined with a cylindrical voxel partition strategy, which inherently handles varying point densities better than standard grids. To counteract the performance drop on clean data typically caused by cylindrical partitioning, they implement a teacher-student knowledge distillation framework (using a grid-voxel teacher) and fine-tune the model using self-training with pseudo-labels generated via test-time augmentation.
Datasets
The authors introduce two new corruption benchmarks derived from popular autonomous driving datasets:
- SemanticKITTI-C (modifying 4,071 validation scans)
- SemanticPOSS-C (modifying 500 validation scans)
Each dataset simulates 16 distinct out-of-domain corruptions across three primary groups and multiple intensity levels:
- Adverse Weather: Physics-based simulations of fog (varying the backscattering coefficient $\beta$) and snowfall (varying accumulation rates).
- Measurement Noise: Global outliers (uniformly distributed noise points) and local distortion (Gaussian jitter applied to point coordinates).
- Cross-Device Discrepancy: Downsampling 64-beam LiDAR data into 32-beam and 16-beam configurations using K-Means clustering on zenith angles to mimic different hardware specs.
Key Insights and Ideas
- Input Representation Dictates Vulnerability: The choice of how 3D data is represented fundamentally alters its robustness profile. Projection-based methods are highly sensitive to weather conditions like fog because spatial noise corrupts the structural integrity of 2D range images. Conversely, they are remarkably resilient to local coordinate jittering.
- The Hybrid-Representation Trap: Modern state-of-the-art models often combine points, voxels, and projections to maximize clean-data accuracy. However, the authors discover that these hybrid architectures severely compromise robustness, as corruptions in a single representation branch propagate and degrade the entire network.
- Point Aggregation vs. Spatial Noise: Point-based models handle global weather noise well but fail under local distortions because coordinate jittering disrupts the local geometric aggregation functions (like attentive pooling) they rely on.
- Voxel Superiority: Pure voxel-based architectures provide the most balanced all-around robustness, particularly when using cylindrical partitioning instead of traditional Cartesian grids.
Key Results
- Baseline Degradation: Projection-based models suffered the most, retaining only about 70-75% of their relative performance (mR) under corrupted conditions.
- Voxel Resilience: Pure MinkowskiNet demonstrated exceptional stability against global noise, retaining 95.8% of its baseline performance even when flooded with 50% global outliers. It also led cross-device generalizability, preserving 76.0% performance on 16-beam downsampled data.
- RLSeg Performance: The proposed RLSeg model achieved top-tier robustness, scoring a Robustness mIoU (RmIoU) of 62.5% on SemanticKITTI-C and 50.1% on SemanticPOSS-C. It maintained 99.9% relative performance under global outliers and 97.8% under local distortion, while simultaneously boosting clean data mIoU to 73.5%.
Limitations and Future Work
A notable limitation identified in the study involves data augmentation trade-offs. While popular 3D data augmentation strategies like Mix3D significantly improve a model's resilience to weather and spatial noise, they actually decrease robustness against cross-device beam downsampling. Because Mix3D creates artificially dense point clouds during training, the model becomes highly vulnerable when later exposed to sparse, low-beam sensor data. The authors note that developing adaptive augmentation techniques capable of generalizing across varying sensor resolutions remains an important direction for future work.
Summary updated