Walking Your LiDOG: A Journey Through Multiple Domains for LiDAR Semantic Segmentation

Cristiano Saltori, Aljoša Ošep, Elisa Ricci, Laura Leal-Taixé

TL;DR

This paper introduces the first experimental benchmark for Domain Generalization in LiDAR Semantic Segmentation (DG-LSS), tackling the problem of deploying models on completely unseen environments and sensors. To bridge the performance gap caused by domain shifts, the authors propose LiDOG, a method that uses a 2D Bird's-Eye-View (BEV) semantic prediction auxiliary task during training to force a 3D network to learn robust, domain-invariant features without adding any computational overhead during inference.

What Problem Does This Paper Address?

LiDAR Semantic Segmentation (LSS) is crucial for autonomous vehicles and mobile robots to understand their surroundings. While state-of-the-art LSS models perform exceptionally well when trained and tested on the same dataset, their performance drops drastically when deployed in a new environment or with a different sensor setup (known as a domain shift).

Existing solutions often rely on Unsupervised Domain Adaptation (UDA), which requires access to unlabeled data from the target domain during training. However, in real-world robotics, a robot must be able to navigate safely in entirely novel environments without retraining. This paper addresses Domain Generalization (DG), where a model is trained purely on source data and must generalize to completely unseen target domains without any prior exposure.

Figure 1: Domain Generalization for LiDAR Semantic Segmentation (DG-LSS). Left: Existing LSS methods are trained and evaluated on point clouds drawn from the same domain. Right: We focus on studying LSS under domain shifts, where the test samples are drawn from a different data distribution. Our paper aims to address the generalization aspect of this task.

Methods

The authors propose LiDOG (LiDAR DOmain Generalization), which augments a standard 3D sparse-convolutional encoder-decoder network with a dense 2D convolutional decoder branch.

The network architecture operates via two parallel branches during training:

  1. 3D Segmentation Branch (Main Task): The input point cloud is voxelized and processed by a 3D sparse-convolutional backbone (MinkowskiNet) to predict voxel-wise semantic labels.
  2. Dense Auxiliary Branch (Proxy Task): The sparse 3D features from the decoder are projected along the vertical (height) axis onto a 2D Bird's-Eye-View (BEV) plane. A standard 2D convolutional head then learns to predict a dense 2D semantic layout of the scene.

Both branches are optimized jointly using a soft DICE loss. Crucially, the 2D BEV auxiliary head is discarded during inference. It serves solely as a regularizer during training to guide the 3D backbone toward learning domain-agnostic features.

LiDOG Architecture Overview

Datasets

To establish their DG-LSS benchmark, the authors map multiple datasets to a unified 7-class vocabulary (vehicle, person, road, sidewalk, terrain, manmade, and vegetation):

  • Synthetic Datasets: Synth4D (generated via the CARLA simulator), featuring two distinct sensor emulations: Synth4D-KITTI (64-beam) and Synth4D-nuScenes (32-beam).
  • Real-World Datasets:
  • SemanticKITTI: High-resolution, 64-beam Velodyne scans captured in Karlsruhe, Germany.
  • nuScenes: Sparser, 32-beam Velodyne scans captured in Boston, USA, and Singapore.

Key Insights and Ideas

  • The BEV Plane as a Geometric Equalizer: LiDAR point clouds vary significantly across domains due to differing beam counts, ring patterns, and sensor placements. However, when projected down to a 2D BEV plane, the structural layouts of cities (roads, sidewalks, buildings) look geometrically much more similar across domains.
  • Implicit Regularization: Forcing the 3D network to succeed at the 2D BEV prediction task implicitly penalizes it for relying on sensor-specific artifacts (like specific vertical beam angles). This encourages the network to focus on robust, transferable spatial structures.
  • Inference-Time Efficiency: Unlike many domain generalization techniques that introduce complex modules or test-time augmentations, LiDOG achieves its gains purely through training-time regularization, introducing zero extra parameters or latency at deployment.
BEV projections make different domains look geometrically more similar

Key Results

LiDOG consistently outperforms existing 3D data augmentations (e.g., Mix3D, PointCutMix) and image-based domain generalization methods adapted to 3D data:

  • Real-to-Real Generalization (SemanticKITTI $\rightarrow$ nuScenes): A baseline model trained only on the source domain drops to 26.53 mIoU on the target. LiDOG recovers a significant portion of this gap, achieving 34.88 mIoU (+8.35 improvement).
  • Real-to-Real Generalization (nuScenes $\rightarrow$ SemanticKITTI): LiDOG improves the baseline from 29.55 mIoU to 41.22 mIoU (+11.67 improvement).
  • Synthetic-to-Real Generalization: When trained on Synth4D-KITTI and tested on the real SemanticKITTI dataset, LiDOG boosts performance from 24.69 mIoU (baseline) to 44.18 mIoU (+19.49 improvement).

Limitations and Future Work

The primary limitation of LiDOG stems from its core strength: the BEV projection collapses the vertical axis. When multiple semantic classes overlap vertically—such as a grassy terrain directly underneath a tree canopy, or a sidewalk under a building overhang—the projection mixes these labels. This vertical collapse leads to sub-optimal performance on specific classes like terrain.

The authors suggest that future iterations could mitigate this by introducing soft labels for vertically overlapping classes or by utilizing multiple BEV height slices to preserve coarse vertical structure.

Summary updated

Abstract

The ability to deploy robots that can operate safely in diverse environments is crucial for developing embodied intelligent agents. As a community, we have made tremendous progress in within-domain LiDAR semantic segmentation. However, do these methods generalize across domains? To answer this question, we design the first experimental setup for studying domain generalization (DG) for LiDAR semantic segmentation (DG-LSS). Our results confirm a significant gap between methods, evaluated in a cross-domain setting: for example, a model trained on the source dataset (SemanticKITTI) obtains $26.53$ mIoU on the target data, compared to $48.49$ mIoU obtained by the model trained on the target domain (nuScenes). To tackle this gap, we propose the first method specifically designed for DG-LSS, which obtains $34.88$ mIoU on the target domain, outperforming all baselines. Our method augments a sparse-convolutional encoder-decoder 3D segmentation network with an additional, dense 2D convolutional decoder that learns to classify a birds-eye view of the point cloud. This simple auxiliary task encourages the 3D network to learn features that are robust to sensor placement shifts and resolution, and are transferable across domains. With this work, we aim to inspire the community to develop and evaluate future models in such cross-domain conditions.
BibTeX
@inproceedings{saltori2023walking,
  author = {Cristiano Saltori and Aljoša Ošep and Elisa Ricci and Laura Leal-Taixé},
  title = {{Walking Your LiDOG: A Journey Through Multiple Domains for LiDAR Semantic Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure