Few-shot 3D LiDAR Semantic Segmentation for Autonomous Driving

Jilin Mei, Junbao Zhou, Yu Hu

Abstract: In autonomous driving, the novel objects and lack of annotations challenge the traditional 3D LiDAR semantic segmentation based on deep learning. Few-shot learning is a feasible way to solve these issues. However, currently few-shot semantic segmentation methods focus on camera data, and most of them only predict the novel classes without considering the base classes. This setting cannot be directly applied to autonomous driving due to safety concerns. Thus, we propose a few-shot 3D LiDAR semantic segmentation method that predicts both novel classes and base classes simultaneously. Our method tries to solve the background ambiguity problem in generalized few-shot semantic segmentation. We first review the original cross-entropy and knowledge distillation losses, then propose a new loss function that incorporates the background information to achieve 3D LiDAR few-shot semantic segmentation. Extensive experiments on SemanticKITTI demonstrate the effectiveness of our method.

1. Introduction

Traditional deep learning models for 3D LiDAR semantic segmentation are notoriously data-hungry, requiring thousands of meticulously annotated scans to achieve high accuracy. In the dynamic environment of autonomous driving, encountering novel object classes—such as specific types of construction vehicles or rare road obstacles—is inevitable. While Few-Shot Learning (FSL) offers a pathway to learn from limited examples (typically 1–100), existing research predominantly focuses on 2D image data and "novel-only" prediction.

In a safety-critical autonomous driving context, a model that learns a new class but forgets how to identify the "road" or "sidewalk" (base classes) is unusable. This research addresses Generalized Few-Shot Semantic Segmentation (GFSS), where the model must maintain performance on original base classes while simultaneously learning to segment novel classes. The authors identify "background ambiguity" as the primary obstacle: in few-shot training, the background of a novel sample often contains base class objects that are incorrectly labeled as generic background, leading to catastrophic forgetting.

2. Methodology

The proposed framework utilizes a three-step transfer learning approach built upon the SalsaNext backbone. To handle the inherent sparsity of LiDAR data, the system employs Range-View Projection, transforming 3D points into a $64 \times 2048 \times 5$ 2D range image to leverage efficient convolutional neural networks.

2.1 Addressing Background Ambiguity

The core innovation lies in the loss functions designed for the fine-tuning stage:

  • Unbiased Cross-Entropy Loss ($L_{ce}^2$): Standard cross-entropy treats all non-novel pixels as a single background class. This method instead uses the original base model to predict the semantics of these background points. By recognizing that the background is semantically dense, the model avoids "unlearning" base classes present in the new training samples.
  • Knowledge Distillation ($L_{kd}$): A teacher-student architecture is employed where the original base model (teacher) guides the fine-tuned model (student). This ensures the student's probability distributions for base classes remain consistent with the teacher's prior knowledge.
  • Lovász-Softmax Loss: To align the training objective with the evaluation metric, the authors incorporate Lovász-Softmax to optimize the Mean Intersection over Union (mIoU) directly.

3. Experimental Results

The method was evaluated using the SemanticKITTI dataset, which includes 43,000 LiDAR scans. The experiments defined 16 base classes (e.g., road, vegetation) and 4 novel classes (car, person, bicyclist, motorcyclist) across various "shot" counts.

3.1 Performance Metrics

In the 100-shot scenario, the proposed method achieved a total mIoU of 55.5%, significantly outperforming established baselines such as GFSS-dyn (53.0%) and Learning without Forgetting (LwF) (53.9%).

Crucially, the model demonstrated high stability in its original knowledge. The base class performance ($mIoU_b$) only dropped from 58.7% (original model) to 56.6%, while the novel class performance ($mIoU_n$) reached 50.9%. This indicates that the combination of unbiased loss and knowledge distillation effectively mitigates catastrophic forgetting.

4. Discussion and Key Insights

The research highlights that in LiDAR-based autonomous driving, the "background" is rarely empty space; it is filled with critical semantic information. By treating the background as a known entity rather than an ignored void, the model maintains a safety-first design. Knowledge distillation acts as a stabilizer, preventing the neural network weights from drifting too far during the adaptation to new classes.

5. Conclusion and Future Work

This paper provides a robust framework for generalized few-shot learning in 3D LiDAR segmentation. While the results on SemanticKITTI are promising, future research should explore the method's performance across different sensor architectures (e.g., Cylinder3D) and address domain shift—the challenge of applying few-shot learning when the new data comes from a different geographic location or sensor type. Additionally, moving toward "on-the-fly" weight updates during active driving remains a significant frontier for the field.

Summary updated

Abstract

In autonomous driving, the novel objects and lack of annotations challenge the traditional 3D LiDAR semantic segmentation based on deep learning. Few-shot learning is a feasible way to solve these issues. However, currently few-shot semantic segmentation methods focus on camera data, and most of them only predict the novel classes without considering the base classes. This setting cannot be directly applied to autonomous driving due to safety concerns. Thus, we propose a few-shot 3D LiDAR semantic segmentation method that predicts both novel classes and base classes simultaneously. Our method tries to solve the background ambiguity problem in generalized few-shot semantic segmentation. We first review the original cross-entropy and knowledge distillation losses, then propose a new loss function that incorporates the background information to achieve 3D LiDAR few-shot semantic segmentation. Extensive experiments on SemanticKITTI demonstrate the effectiveness of our method.
BibTeX
@inproceedings{mei2023few,
  author = {Jilin Mei and Junbao Zhou and Yu Hu},
  title = {{Few-shot 3D LiDAR Semantic Segmentation for Autonomous Driving}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=bbLpPihWk4},
}
Figure