Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation

Shoumeng Qiu, Feng Jiang, Haiqiang Zhang, Xiangyang Xue, Jian Pu

Introduction

In the field of autonomous driving, LiDAR sensors are the primary eyes of the vehicle, providing critical 3D spatial information. However, these sensors face a fundamental physical limitation: point clouds become increasingly sparse as the distance from the sensor increases. This sparsity creates a significant hurdle for semantic segmentation models, particularly when identifying "hard classes"—small or rare objects such as motorcyclists, traffic signs, or pedestrians—which may be represented by only a handful of points.

While fusing multiple sequential LiDAR scans can create a denser point cloud and improve accuracy, the computational cost of processing temporal data is often prohibitive for real-time inference. This paper introduces the Multi-to-Single Knowledge Distillation (M2SKD) framework, a method designed to bridge this gap. M2SKD allows a lightweight single-scan model (the student) to inherit the "wisdom" of a dense, multi-scan model (the teacher) during training, maintaining high inference speeds at test time without the need for temporal data.

Methodology

The M2SKD framework utilizes a teacher-student paradigm where the teacher model is trained on dense, fused point clouds, and the student model is trained on single scans. The core innovation lies in how the data is prepared and how the knowledge is transferred.

Priori-based Sparse Fusion and Alignment

To create an effective teacher, the authors employ Priori-based Sparse Fusion. Rather than fusing every point from sequential scans—which introduces significant noise and computational lag—the framework selectively fuses instances belonging to "hard classes" (those with historically low Intersection over Union, or IoU). This forces the model to focus its learning capacity on the most difficult objects. To ensure these fused points are accurate, Registration-based Alignment is used to precisely align moving objects across frames, preventing the "ghosting" effects that occur with simple global pose translation.

Multilevel Distillation

The student model mimics the teacher across three distinct levels:

  1. Feature Representation: Aligning intermediate layer activations to ensure the student learns similar spatial hierarchies.
  2. Logits: Matching the final probability distributions to capture the teacher's classification logic.
  3. Instance-Aware Affinity Distillation (IAAD): This is the framework's primary technical contribution. Unlike previous methods that calculate relationships between arbitrary "supervoxels" (which often include background noise), IAAD computes a similarity matrix for points specifically within an object instance. This captures the high-level structural "signature" of an object, teaching the student the underlying geometry of sparse classes.

For datasets lacking instance labels for specific categories, the authors implemented a clustering pipeline using Farthest Point Sampling (FPS) to generate pseudo-instance IDs, ensuring the IAAD remains applicable across various data types.

Experimental Results

The researchers evaluated M2SKD on the SemanticKITTI dataset, a benchmark for autonomous driving containing over 43,000 LiDAR scans. The framework was tested as a "plug-in" for two state-of-the-art baselines: PolarNet and Cylinder3D.

Performance Gains

The integration of M2SKD resulted in significant performance boosts across the board:

  • Overall Accuracy: On the SemanticKITTI test set, M2SKD improved Cylinder3D’s mean IoU (mIoU) by 1.7% and PolarNet’s by 3.7%.
  • Hard Class Breakthroughs: The most dramatic improvements were seen in sparse, difficult-to-detect classes. For the motorcyclist class, the method outperformed the Cylinder3D baseline by 22.8%. Similarly, the bicyclist class saw an 11.2% improvement over the PolarNet baseline.

Efficiency

Crucially, because the distillation process occurs entirely during the training phase, the student model's architecture remains unchanged. This allows the model to maintain its original inference speed, providing the accuracy of a multi-scan system with the efficiency of a single-scan approach.

Conclusion and Future Work

M2SKD demonstrates that temporal information can be treated as a "privileged" signal—valuable for supervision during training but unnecessary for deployment. By focusing on instance-aware structural integrity and selective fusion of hard classes, the framework significantly advances the state of 3D semantic segmentation for autonomous systems.

While the method currently relies on the quality of point cloud registration, future research may explore extending this multi-to-single distillation logic to other 3D tasks, such as object detection, where maintaining temporal consistency is vital but computationally expensive.

Summary updated

BibTeX
@inproceedings{qiu2023multi,
  author = {Shoumeng Qiu and Feng Jiang and Haiqiang Zhang and Xiangyang Xue and Jian Pu},
  title = {{Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=XXAMXyKsAX},
}
Figure