Abstract
ProtoTransfer is a novel framework designed to enhance 3D point cloud segmentation by transferring rich semantic knowledge from 2D images to a LiDAR-only model. Traditional knowledge distillation methods for this task are limited by a "matching bottleneck," where only perfectly aligned point-pixel pairs contribute to learning, leaving the majority of data underutilized. ProtoTransfer overcomes this by employing a class-wise prototype bank as a semantic bridge. This approach allows every LiDAR point to align with representative class features, regardless of direct pixel correspondence. Furthermore, it leverages unmatched image pixels through a pseudo-labeling scheme to enrich the prototype bank. ProtoTransfer achieves state-of-the-art results on nuScenes and SemanticKITTI benchmarks while maintaining real-time inference speeds by discarding the image-related modules after training.
Introduction
In the context of autonomous driving, LiDAR sensors provide essential 3D geometric information but lack the dense semantic cues (color, texture) available in 2D images. While multi-modal fusion models achieve high accuracy by combining both sensors, their computational complexity often precludes real-time deployment. Knowledge distillation offers a solution by training a single-modal "student" (LiDAR) using a multi-modal "teacher."
However, existing distillation methods face two primary challenges:
- The Matching Bottleneck: Knowledge transfer is typically restricted to LiDAR points that have a direct spatial correspondence with image pixels. In datasets like SemanticKITTI, this accounts for only ~16% of points.
- Underutilized Data: Millions of image pixels that do not overlap with LiDAR points are ignored, despite containing valuable semantic information.
Methodology
ProtoTransfer introduces a paradigm shift from strict point-to-pixel distillation to a prototype-based transfer mechanism.
The Prototype Bank
The core of the framework is a class-wise prototype bank—a set of representative feature vectors for each semantic category (e.g., car, pedestrian). These prototypes are initially populated and continuously updated using fused multi-modal features ($F^{fuse}$).
Cross-Modal Transfer and Pseudo-Labeling
During training, every LiDAR point is encouraged to align its features with the corresponding class prototype in the bank. This decouples the learning process from physical sensor alignment, enabling "all-point" supervision. To further improve the prototypes, the model employs a pseudo-labeling scheme for image pixels that lack LiDAR matches ($F^{2d\_u}$). These unmatched features are incorporated into the bank, ensuring that no valuable image data is wasted.
Inference Efficiency
A key advantage of ProtoTransfer is its efficiency. Once the training phase is complete, the image backbone, fusion modules, and prototype bank are discarded. The final model performs inference using only the LiDAR backbone, ensuring high-speed operation suitable for real-time applications.
Experimental Results
The model was rigorously evaluated on the nuScenes and SemanticKITTI benchmarks.
- nuScenes: ProtoTransfer achieved 82.1% mIoU on the test set, ranking 2nd on the Lidarseg challenge leaderboard at the time of publication.
- SemanticKITTI: It reached 73.6% mIoU, outperforming the previous state-of-the-art distillation method, 2DPASS, by 0.7%.
- Efficiency: The model maintains a low latency of 44ms, significantly faster than dual-modal fusion models.
- Qualitative Performance: Visual analysis indicates superior segmentation of object boundaries and improved detection of small or distant objects, such as traffic cones, where traditional point-pixel matching often fails.
Conclusion
ProtoTransfer demonstrates that using prototypes as a semantic bridge effectively addresses the data alignment and utilization issues in cross-modal distillation. By leveraging both matched and unmatched multi-modal data during training, it produces a high-performance LiDAR-only model. Future work may explore extending this prototype-based transfer to related tasks like 3D object detection or domain adaptation.
Summary updated