ELiTe: Efficient Image-to-LiDAR Knowledge Transfer for Semantic Segmentation

Zhibo Zhang, Ximing Yang, Weizhong Zhang, Cheng Jin

TL;DR

This paper introduces ELiTe, a novel framework for efficient image-to-LiDAR knowledge transfer to improve semantic segmentation in autonomous driving. By leveraging a powerful Vision Foundation Model (VFM) and a new pseudo-labeling strategy, ELiTe achieves state-of-the-art performance with significantly fewer parameters and real-time inference speeds, addressing key challenges in cross-modal learning for LiDAR.

What Problem Does This Paper Address?

Semantic segmentation of LiDAR point clouds is crucial for autonomous driving. While cross-modal knowledge transfer from images can enhance this task, existing methods face several challenges. Firstly, car-mounted camera images often lack diversity and are repetitive, making it difficult to train robust teacher models. Secondly, the ground truth labels for LiDAR data are often sparse and inaccurate, leading to weak supervision for teacher models. Finally, many effective cross-modal methods result in large, computationally expensive models, hindering real-time applications. ELiTe aims to overcome these limitations by enabling efficient and effective knowledge transfer from a powerful, pre-trained image model to a lightweight LiDAR segmentation student model.

Methods

ELiTe is built upon three core components:

  • VFM Teacher and Lightweight Student: The framework utilizes a Vision Foundation Model (VFM), specifically the Segment Anything Model (SAM), as the teacher. This VFM is fine-tuned using Parameter-Efficient Fine-Tuning (PEFT) to adapt it to the domain while minimizing computational costs. A lightweight student network is responsible for LiDAR semantic segmentation.
  • Patch-to-Point Multi-Stage Knowledge Distillation (PPMSKD): This novel distillation technique transfers knowledge from the VFM teacher to the student across multiple stages. It aligns image features with point cloud features, ensuring improved representation learning for the student. The distillation process involves extracting features from both image patches and LiDAR points and using a KL divergence loss to guide the transfer.
  • SAM-based Pseudo-Label Generation (SAM-PLG): To address the issue of sparse and inaccurate ground truth labels, ELiTe employs SAM-PLG. This two-stage process generates dense and accurate pseudo-labels from sparse input. It leverages SAM's ability to segment objects based on prompts and uses semantic and instance information to refine these masks, ultimately providing stronger supervision for the teacher model.

Datasets

  • SemanticKITTI: This is the primary dataset used for training and evaluation. It's a large-scale outdoor benchmark for LiDAR semantic segmentation. The dataset is split into training, validation (sequence 08), and testing (sequences 11-21) sets.
  • KITTI: Used in conjunction with SemanticKITTI, the KITTI dataset provides the raw image data for the experiments.
  • SAM Dataset: The Segment Anything Model (SAM) itself is trained on a massive dataset of approximately 11 million images, which contributes to its powerful zero-shot capabilities.

Key Insights and Ideas

  • Leveraging Pre-trained Foundation Models: The paper demonstrates the significant advantage of using large, pre-trained Vision Foundation Models (like SAM) as teachers, even when fine-tuned with parameter-efficient methods. This allows for effective knowledge transfer from diverse, open-world image data.
  • Pseudo-Labeling for Weak Supervision: SAM-PLG is a crucial innovation that effectively transforms weak, sparse LiDAR labels into strong, dense supervision. This significantly boosts the teacher model's ability to learn robust semantic representations.
  • Efficiency is Key: ELiTe prioritizes efficiency by using PEFT for teacher fine-tuning and a lightweight student model. This allows for real-time inference speeds, which is critical for autonomous driving applications, while maintaining competitive performance.
  • Teacher Modality Matters: The study highlights that simply fusing 2D and 3D teachers doesn't always guarantee better student performance. The quality and adaptability of the teacher's features, as facilitated by ELiTe's approach, are more critical than just the modality.

Key Results

  • State-of-the-Art Performance: ELiTe achieves state-of-the-art results on the SemanticKITTI benchmark, outperforming existing real-time inference models.
  • Real-time Inference: The proposed method achieves a speed of 24Hz, significantly surpassing other real-time inference methods.
  • Parameter Efficiency: ELiTe uses a lightweight student model with only 1.9M trainable parameters, making it highly efficient compared to many other methods.
  • Significant Improvement: Compared to the baseline 2DPASS-base, ELiTe shows a substantial improvement in mIoU (71.4% vs. 67.7%).
  • Ablation Studies: Ablation studies confirm the effectiveness of each component, with PPMSKD contributing a significant performance boost and SAM-PLG further enhancing results.

Limitations and Future Work

  • Teacher Feature Mimicry: The paper notes that the student's feature clustering doesn't precisely replicate the teacher's, suggesting that vanilla knowledge distillation might not fully facilitate effective cross-modal transfer. This indicates a potential area for future research in improving the fidelity of knowledge transfer.
  • Pseudo-Label Imperfections: While SAM-PLG generates high-quality pseudo-labels, there are occasional gaps in density and minor imprecisions in segmentation boundaries. These are attributed to inherent inaccuracies in the initial ground truth and a trade-off between efficiency and absolute precision.
  • Future Directions: The paper implicitly suggests that further research into more sophisticated cross-modal distillation techniques and exploring even more powerful foundation models could lead to further improvements. The focus on efficiency also opens avenues for deploying such models in resource-constrained environments.

Summary updated

BibTeX
@inproceedings{zhang2024elite,
  author = {Zhibo Zhang and Ximing Yang and Weizhong Zhang and Cheng Jin},
  title = {{ELiTe: Efficient Image-to-LiDAR Knowledge Transfer for Semantic Segmentation}},
  booktitle = {ICME},
  year = {2024},
  doi = {10.1109/icme57554.2024.10687769},
}
Figure