ProtoTransfer: Cross-Modal Prototype Transfer for Point Cloud Segmentation

Pin Tang, Hai-Ming Xu, Chao MaICCV 2023 · DOI

Abstract

ProtoTransfer is a novel framework designed to enhance 3D point cloud segmentation by transferring rich semantic knowledge from 2D images to a LiDAR-only model. Traditional knowledge distillation methods for this task are limited by a "matching bottleneck," where only perfectly aligned point-pixel pairs contribute to learning, leaving the majority of data underutilized. ProtoTransfer overcomes this by employing a class-wise prototype bank as a semantic bridge. This approach allows every LiDAR point to align with representative class features, regardless of direct pixel correspondence. Furthermore, it leverages unmatched image pixels through a pseudo-labeling scheme to enrich the prototype bank. ProtoTransfer achieves state-of-the-art results on nuScenes and SemanticKITTI benchmarks while maintaining real-time inference speeds by discarding the image-related modules after training.

Introduction

In the context of autonomous driving, LiDAR sensors provide essential 3D geometric information but lack the dense semantic cues (color, texture) available in 2D images. While multi-modal fusion models achieve high accuracy by combining both sensors, their computational complexity often precludes real-time deployment. Knowledge distillation offers a solution by training a single-modal "student" (LiDAR) using a multi-modal "teacher."

However, existing distillation methods face two primary challenges:

  1. The Matching Bottleneck: Knowledge transfer is typically restricted to LiDAR points that have a direct spatial correspondence with image pixels. In datasets like SemanticKITTI, this accounts for only ~16% of points.
  2. Underutilized Data: Millions of image pixels that do not overlap with LiDAR points are ignored, despite containing valuable semantic information.
Comparison of ProtoTransfer with traditional distillation methods.

Methodology

ProtoTransfer introduces a paradigm shift from strict point-to-pixel distillation to a prototype-based transfer mechanism.

The Prototype Bank

The core of the framework is a class-wise prototype bank—a set of representative feature vectors for each semantic category (e.g., car, pedestrian). These prototypes are initially populated and continuously updated using fused multi-modal features ($F^{fuse}$).

Cross-Modal Transfer and Pseudo-Labeling

During training, every LiDAR point is encouraged to align its features with the corresponding class prototype in the bank. This decouples the learning process from physical sensor alignment, enabling "all-point" supervision. To further improve the prototypes, the model employs a pseudo-labeling scheme for image pixels that lack LiDAR matches ($F^{2d\_u}$). These unmatched features are incorporated into the bank, ensuring that no valuable image data is wasted.

The ProtoTransfer architecture overview.

Inference Efficiency

A key advantage of ProtoTransfer is its efficiency. Once the training phase is complete, the image backbone, fusion modules, and prototype bank are discarded. The final model performs inference using only the LiDAR backbone, ensuring high-speed operation suitable for real-time applications.

Experimental Results

The model was rigorously evaluated on the nuScenes and SemanticKITTI benchmarks.

  • nuScenes: ProtoTransfer achieved 82.1% mIoU on the test set, ranking 2nd on the Lidarseg challenge leaderboard at the time of publication.
  • SemanticKITTI: It reached 73.6% mIoU, outperforming the previous state-of-the-art distillation method, 2DPASS, by 0.7%.
  • Efficiency: The model maintains a low latency of 44ms, significantly faster than dual-modal fusion models.
  • Qualitative Performance: Visual analysis indicates superior segmentation of object boundaries and improved detection of small or distant objects, such as traffic cones, where traditional point-pixel matching often fails.

Conclusion

ProtoTransfer demonstrates that using prototypes as a semantic bridge effectively addresses the data alignment and utilization issues in cross-modal distillation. By leveraging both matched and unmatched multi-modal data during training, it produces a high-performance LiDAR-only model. Future work may explore extending this prototype-based transfer to related tasks like 3D object detection or domain adaptation.

Summary updated

BibTeX
@inproceedings{tang2023prototransfer,
  author = {Pin Tang and Hai-Ming Xu and Chao Ma},
  title = {{ProtoTransfer: Cross-Modal Prototype Transfer for Point Cloud Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00309},
}
Figure