TL;DR
This paper introduces the first experimental benchmark for Domain Generalization in LiDAR Semantic Segmentation (DG-LSS), tackling the problem of deploying models on completely unseen environments and sensors. To bridge the performance gap caused by domain shifts, the authors propose LiDOG, a method that uses a 2D Bird's-Eye-View (BEV) semantic prediction auxiliary task during training to force a 3D network to learn robust, domain-invariant features without adding any computational overhead during inference.
What Problem Does This Paper Address?
LiDAR Semantic Segmentation (LSS) is crucial for autonomous vehicles and mobile robots to understand their surroundings. While state-of-the-art LSS models perform exceptionally well when trained and tested on the same dataset, their performance drops drastically when deployed in a new environment or with a different sensor setup (known as a domain shift).
Existing solutions often rely on Unsupervised Domain Adaptation (UDA), which requires access to unlabeled data from the target domain during training. However, in real-world robotics, a robot must be able to navigate safely in entirely novel environments without retraining. This paper addresses Domain Generalization (DG), where a model is trained purely on source data and must generalize to completely unseen target domains without any prior exposure.
Methods
The authors propose LiDOG (LiDAR DOmain Generalization), which augments a standard 3D sparse-convolutional encoder-decoder network with a dense 2D convolutional decoder branch.
The network architecture operates via two parallel branches during training:
- 3D Segmentation Branch (Main Task): The input point cloud is voxelized and processed by a 3D sparse-convolutional backbone (MinkowskiNet) to predict voxel-wise semantic labels.
- Dense Auxiliary Branch (Proxy Task): The sparse 3D features from the decoder are projected along the vertical (height) axis onto a 2D Bird's-Eye-View (BEV) plane. A standard 2D convolutional head then learns to predict a dense 2D semantic layout of the scene.
Both branches are optimized jointly using a soft DICE loss. Crucially, the 2D BEV auxiliary head is discarded during inference. It serves solely as a regularizer during training to guide the 3D backbone toward learning domain-agnostic features.
Datasets
To establish their DG-LSS benchmark, the authors map multiple datasets to a unified 7-class vocabulary (vehicle, person, road, sidewalk, terrain, manmade, and vegetation):
- Synthetic Datasets: Synth4D (generated via the CARLA simulator), featuring two distinct sensor emulations: Synth4D-KITTI (64-beam) and Synth4D-nuScenes (32-beam).
- Real-World Datasets:
- SemanticKITTI: High-resolution, 64-beam Velodyne scans captured in Karlsruhe, Germany.
- nuScenes: Sparser, 32-beam Velodyne scans captured in Boston, USA, and Singapore.
Key Insights and Ideas
- The BEV Plane as a Geometric Equalizer: LiDAR point clouds vary significantly across domains due to differing beam counts, ring patterns, and sensor placements. However, when projected down to a 2D BEV plane, the structural layouts of cities (roads, sidewalks, buildings) look geometrically much more similar across domains.
- Implicit Regularization: Forcing the 3D network to succeed at the 2D BEV prediction task implicitly penalizes it for relying on sensor-specific artifacts (like specific vertical beam angles). This encourages the network to focus on robust, transferable spatial structures.
- Inference-Time Efficiency: Unlike many domain generalization techniques that introduce complex modules or test-time augmentations, LiDOG achieves its gains purely through training-time regularization, introducing zero extra parameters or latency at deployment.
Key Results
LiDOG consistently outperforms existing 3D data augmentations (e.g., Mix3D, PointCutMix) and image-based domain generalization methods adapted to 3D data:
- Real-to-Real Generalization (SemanticKITTI $\rightarrow$ nuScenes): A baseline model trained only on the source domain drops to 26.53 mIoU on the target. LiDOG recovers a significant portion of this gap, achieving 34.88 mIoU (+8.35 improvement).
- Real-to-Real Generalization (nuScenes $\rightarrow$ SemanticKITTI): LiDOG improves the baseline from 29.55 mIoU to 41.22 mIoU (+11.67 improvement).
- Synthetic-to-Real Generalization: When trained on Synth4D-KITTI and tested on the real SemanticKITTI dataset, LiDOG boosts performance from 24.69 mIoU (baseline) to 44.18 mIoU (+19.49 improvement).
Limitations and Future Work
The primary limitation of LiDOG stems from its core strength: the BEV projection collapses the vertical axis. When multiple semantic classes overlap vertically—such as a grassy terrain directly underneath a tree canopy, or a sidewalk under a building overhang—the projection mixes these labels. This vertical collapse leads to sub-optimal performance on specific classes like terrain.
The authors suggest that future iterations could mitigate this by introducing soft labels for vertically overlapping classes or by utilizing multiple BEV height slices to preserve coarse vertical structure.
Summary updated