LidarCLIP or: How I Learned To Talk to Point Clouds

Georg Hess, Adam Tonderski, Christoffer Petersson, Kalle Åström, Lennart SvenssonWACV 2024

TL;DR

This paper introduces LidarCLIP, a novel method that bridges the gap between natural language and 3D lidar point clouds by mapping lidar data into the established CLIP embedding space. This allows for powerful zero-shot understanding and retrieval of lidar scenes using text, significantly outperforming previous approaches and enabling new applications like point cloud captioning.

What Problem Does This Paper Address?

The success of models like CLIP in connecting natural language to images has revolutionized how we interact with visual data. However, extending this capability to other crucial sensor modalities, like lidar point clouds, has been hindered by a lack of large-scale, paired text-lidar datasets. This limitation restricts the development of advanced NLP-driven applications for 3D perception, which is vital for fields like autonomous driving. Existing attempts to bridge this gap have often been application-specific, limited to synthetic data, or required extensive manual annotations, failing to leverage the vast amounts of readily available automotive data.

Methods

LidarCLIP tackles this challenge by leveraging existing image-lidar datasets, which are more common in the automotive domain. The core idea is to train a lidar encoder to mimic the embeddings produced by a pre-trained, frozen CLIP image encoder. This is achieved by using image-lidar pairs from datasets like ONCE. The image encoder acts as an intermediary, allowing the lidar encoder to learn a representation that aligns with CLIP's rich semantic space.

Specifically, the method involves:

  • Lidar Encoder: A new encoder (e.g., a Sparse Transformer) is trained to process raw lidar point clouds.
  • Frozen CLIP Image Encoder: The powerful, pre-trained CLIP image encoder is used as a "teacher" to provide target embeddings.
  • Image-Lidar Alignment: Point clouds are transformed into the camera's coordinate system, and points outside the image's field of view are removed. This ensures that the lidar data corresponds to what's visible in the paired image.
  • Training Objective: A Mean Squared Error (MSE) loss is used to minimize the distance between the lidar embedding and the corresponding image embedding in the CLIP space. This approach is chosen over contrastive loss to avoid large batch size requirements and focus on mapping to an existing space.

Datasets

  • ONCE Dataset: Used for both training and evaluation. This large-scale automotive dataset contains millions of image-lidar pairs.
  • nuScenes Dataset: Used for evaluating domain transfer capabilities. This dataset has different lidar sensor characteristics and is collected in different geographical locations than ONCE.

Key Insights and Ideas

  • Image as a Bridge: The paper elegantly demonstrates that images can serve as a powerful intermediary to connect text and lidar data, bypassing the need for direct text-lidar pairs.
  • Complementary Strengths: LidarCLIP, when combined with standard image-based CLIP, offers superior retrieval performance, especially for challenging scenarios where one modality might fail (e.g., extreme glare in images, or distinguishing objects in lidar).
  • Leveraging Existing Models: The approach effectively transfers the rich semantic understanding of CLIP to the 3D domain without requiring extensive retraining of the CLIP model itself.
  • Zero-Shot Power for 3D: LidarCLIP enables zero-shot classification and retrieval for lidar data, opening up new possibilities for understanding 3D scenes with natural language.

Key Results

  • Zero-Shot Classification: LidarCLIP significantly outperforms previous methods like PointCLIP and CLIP2Point on the ONCE dataset, achieving top-1 accuracies of 60.8% for objects and 73.3% for overall scenes, compared to ~25% for prior lidar-specific methods.
  • Retrieval Performance:
    • LidarCLIP-only retrieval achieves strong results, outperforming image-only retrieval in several categories.
  • Joint retrieval (combining image and lidar features) consistently yields the best performance across various categories, with an overall precision of 0.876 at rank 100.
  • The method shows remarkable robustness to domain shift, with minimal performance degradation when transferring from ONCE to nuScenes.
  • Generative Applications: LidarCLIP successfully enables point cloud captioning and lidar-to-image generation with minimal additional training, producing qualitatively good results.

Limitations and Future Work

  • Dataset Diversity: The current training relies on a single automotive dataset (ONCE), which may limit generalization to other domains (e.g., indoor environments) or diverse sensor configurations. Future work could involve training on multiple datasets with varied lidar sensors and scene conditions.
  • Generative Model Realism: While generative applications show promise, the generated images are not entirely photorealistic, partly due to a lack of fine-tuning. The authors suggest that projecting point cloud regions into the image plane during diffusion could improve realism.
  • Captioning Vocabulary: The captions generated for lidar data sometimes lack diversity, likely due to the limited automotive-specific vocabulary in the COCO dataset used for training the captioning model.
  • Future Directions: The authors envision LidarCLIP inspiring further research into connecting text and point cloud understanding, including tasks like referring object detection and open-set semantic segmentation in 3D.

Summary updated

BibTeX
@inproceedings{hess2024lidarclip,
  author = {Georg Hess and Adam Tonderski and Christoffer Petersson and Kalle Åström and Lennart Svensson},
  title = {{LidarCLIP or: How I Learned To Talk to Point Clouds}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure