Benchmarking the Robustness of LiDAR Semantic Segmentation Models

Xu Yan, Chaoda Zheng, Ying Xue, Zhen Li, Shuguang Cui, Dengxin Dai

TL;DR

LiDAR semantic segmentation models achieve impressive accuracy on clean datasets but degrade significantly when deployed in unpredictable real-world environments. This paper introduces the first large-scale robustness benchmarks—SemanticKITTI-C and SemanticPOSS-C—to evaluate 11 state-of-the-art models across 16 real-world corruptions. Leveraging insights from this comprehensive study, the authors design RLSeg, a robust voxel-based model that significantly improves out-of-distribution generalizability without sacrificing clean-data performance.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on LiDAR semantic segmentation to perceive their surroundings and make safety-critical decisions. While current deep learning models excel on standard, pristine validation datasets, real-world deployment exposes them to severe data corruptions. These include adverse weather (fog, snow), sensor measurement noise, and cross-device discrepancies (e.g., running a model trained on a high-resolution 64-beam LiDAR on a vehicle equipped with a cheaper 16-beam sensor).

Prior robustness research in 3D vision has primarily focused on synthetic, single-object datasets like ModelNet40, which fail to capture the complexity of large-scale driving scenes. This paper fills the gap by establishing a rigorous framework to analyze how different input representations, network architectures, and training strategies withstand real-world environmental and hardware variations.

Methods

The authors systematically benchmark 11 existing LiDAR semantic segmentation models, categorizing them by their primary input representations:

  • Projection-based methods: Project 3D point clouds onto 2D planes (Range View or Bird's-Eye View) to leverage efficient 2D CNNs.
  • Point-based methods: Directly process raw, irregular point clouds using localized geometric aggregation functions.
  • Voxel-based methods: Convert point clouds into 3D voxel grids and apply sparse 3D convolutions.

Based on a comprehensive analysis of these architectures, the authors introduce RLSeg (Robust LiDAR Segmentation). RLSeg utilizes a single-representation voxel backbone (MinkowskiNet) combined with a cylindrical voxel partition strategy, which inherently handles varying point densities better than standard grids. To counteract the performance drop on clean data typically caused by cylindrical partitioning, they implement a teacher-student knowledge distillation framework (using a grid-voxel teacher) and fine-tune the model using self-training with pseudo-labels generated via test-time augmentation.

Datasets

The authors introduce two new corruption benchmarks derived from popular autonomous driving datasets:

  • SemanticKITTI-C (modifying 4,071 validation scans)
  • SemanticPOSS-C (modifying 500 validation scans)

Each dataset simulates 16 distinct out-of-domain corruptions across three primary groups and multiple intensity levels:

  1. Adverse Weather: Physics-based simulations of fog (varying the backscattering coefficient $\beta$) and snowfall (varying accumulation rates).
  2. Measurement Noise: Global outliers (uniformly distributed noise points) and local distortion (Gaussian jitter applied to point coordinates).
  3. Cross-Device Discrepancy: Downsampling 64-beam LiDAR data into 32-beam and 16-beam configurations using K-Means clustering on zenith angles to mimic different hardware specs.
Examples of corruptions in the proposed SemanticKITTI-C benchmark.

Key Insights and Ideas

  • Input Representation Dictates Vulnerability: The choice of how 3D data is represented fundamentally alters its robustness profile. Projection-based methods are highly sensitive to weather conditions like fog because spatial noise corrupts the structural integrity of 2D range images. Conversely, they are remarkably resilient to local coordinate jittering.
  • The Hybrid-Representation Trap: Modern state-of-the-art models often combine points, voxels, and projections to maximize clean-data accuracy. However, the authors discover that these hybrid architectures severely compromise robustness, as corruptions in a single representation branch propagate and degrade the entire network.
  • Point Aggregation vs. Spatial Noise: Point-based models handle global weather noise well but fail under local distortions because coordinate jittering disrupts the local geometric aggregation functions (like attentive pooling) they rely on.
  • Voxel Superiority: Pure voxel-based architectures provide the most balanced all-around robustness, particularly when using cylindrical partitioning instead of traditional Cartesian grids.
Visualizing the impact of simulated fog on raw LiDAR point clouds.

Key Results

  • Baseline Degradation: Projection-based models suffered the most, retaining only about 70-75% of their relative performance (mR) under corrupted conditions.
  • Voxel Resilience: Pure MinkowskiNet demonstrated exceptional stability against global noise, retaining 95.8% of its baseline performance even when flooded with 50% global outliers. It also led cross-device generalizability, preserving 76.0% performance on 16-beam downsampled data.
  • RLSeg Performance: The proposed RLSeg model achieved top-tier robustness, scoring a Robustness mIoU (RmIoU) of 62.5% on SemanticKITTI-C and 50.1% on SemanticPOSS-C. It maintained 99.9% relative performance under global outliers and 97.8% under local distortion, while simultaneously boosting clean data mIoU to 73.5%.

Limitations and Future Work

A notable limitation identified in the study involves data augmentation trade-offs. While popular 3D data augmentation strategies like Mix3D significantly improve a model's resilience to weather and spatial noise, they actually decrease robustness against cross-device beam downsampling. Because Mix3D creates artificially dense point clouds during training, the model becomes highly vulnerable when later exposed to sparse, low-beam sensor data. The authors note that developing adaptive augmentation techniques capable of generalizing across varying sensor resolutions remains an important direction for future work.

Summary updated

Abstract

When using LiDAR semantic segmentation models for safety-critical applications such as autonomous driving, it is essential to understand and improve their robustness with respect to a large range of LiDAR corruptions. In this paper, we aim to comprehensively analyze the robustness of LiDAR semantic segmentation models under various corruptions. To rigorously evaluate the robustness and generalizability of current approaches, we propose a new benchmark called SemanticKITTI-C, which features 16 out-of-domain LiDAR corruptions in three groups, namely adverse weather, measurement noise and cross-device discrepancy. Then, we systematically investigate 11 LiDAR semantic segmentation models, especially spanning different input representations (e.g., point clouds, voxels, projected images, and etc.), network architectures and training schemes. Through this study, we obtain two insights: 1) We find out that the input representation plays a crucial role in robustness. Specifically, under specific corruptions, different representations perform variously. 2) Although state-of-the-art methods on LiDAR semantic segmentation achieve promising results on clean data, they are less robust when dealing with noisy data. Finally, based on the above observations, we design a robust LiDAR segmentation model (RLSeg) which greatly boosts the robustness with simple but effective modifications. It is promising that our benchmark, comprehensive analysis, and observations can boost future research in robust LiDAR semantic segmentation for safety-critical applications.
BibTeX
@article{yan2023benchmarking,
  author = {Xu Yan and Chaoda Zheng and Ying Xue and Zhen Li and Shuguang Cui and Dengxin Dai},
  title = {{Benchmarking the Robustness of LiDAR Semantic Segmentation Models}},
  journal = {Int. J. Comput. Vis.},
  year = {2023},
  url = {https://openreview.net/forum?id=otQ3MFvS2c},
}
Figure