ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding

Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, Silvio Savarese

TL;DR

ULIP (Unified Language, Image, and Point cloud) is a pre-training framework that aligns 3D point cloud representations with the powerful, pre-aligned semantic space of vision-language models like CLIP. By leveraging "triplets" of images, text, and 3D data, ULIP significantly boosts the performance of existing 3D backbones and achieves a massive leap in zero-shot 3D classification, outperforming previous methods by nearly 29%.

What Problem Does This Paper Address?

In the world of 2D computer vision, models benefit from massive datasets like ImageNet or the billions of image-text pairs used to train CLIP. In contrast, 3D understanding is hampered by a "data drought." Popular 3D datasets like ShapeNet55 contain only about 52,500 samples across 55 categories—orders of magnitude smaller than their 2D counterparts.

This scarcity makes it difficult for 3D models to generalize to new objects or perform "zero-shot" classification (identifying objects they weren't explicitly trained on). While some researchers have tried projecting 3D objects into 2D depth maps to use 2D models, this often loses critical geometric information. The authors of ULIP ask: Can we instead teach 3D encoders to "speak the same language" as powerful, pre-trained vision-language models?

Methods

The core idea of ULIP is to learn a unified representation space where a 3D point cloud, a 2D image of that object, and a textual description of it all sit close together.

Figure 1. Illustration of ULIP. ULIP improves 3D understand- ing by aligning features from images, texts, and point clouds in the same space. To reduce the demand of 3D data, ULIP lever- ages image and text encoders that are pre-trained with large-scale image-text pairs, and aligns 3D representation to the pre-aligned image-text feature space using a small scale of training triplets.
  1. Creating Triplets: Since large-scale (Image, Text, 3D) datasets don't exist, the authors synthesized them using ShapeNet55. They rendered 2D images from multiple angles for each 3D model and used the dataset's metadata to generate text prompts (e.g., "a point cloud model of a [class]").
  2. Leveraging CLIP: The authors used a pre-trained vision-language model (specifically SLIP, an advanced version of CLIP) and kept its image and text encoders frozen. This preserves the rich semantic knowledge learned from millions of 2D pairs.
  3. Cross-modal Contrastive Learning: A 3D backbone (like PointNet++ or PointMLP) is trained to extract features from the point cloud. The model uses a contrastive loss to pull the 3D features toward the corresponding image and text features in the shared CLIP space.
  4. Backbone Agnostic: ULIP is a pre-training strategy. Once the 3D encoder is "aligned" to the CLIP space, it can be plugged into various downstream tasks, either for zero-shot inference or as a starting point for fine-tuning.
Figure 2. Illustration of our method. The inputs of multimodal pre-training (Left) are a batch of objects represented as triplets (image, text, point cloud). Image and text features are extracted from a pre-trained (frozen) vision and language model such as CLIP, and 3D features are extracted from a 3D encoder. Contrastive losses are applied to align the 3D feature of an object to its image and text features during pre-training. The pre-trained 3D encoders are further fine-tuned in downstream tasks, including standard 3D classification (Top Right) and zero-shot 3D classification (Bottom Right).

Datasets

  • Pre-training: ShapeNet55, a subset of ShapeNet containing ~52.5k CAD models.
  • Evaluation:
    • ModelNet40: A synthetic dataset of 3D CAD models (40 categories).
  • ScanObjectNN: A more challenging dataset consisting of real-world 3D scans, which include background noise and occlusions.
  • Zero-Shot Splits: The authors introduced "Medium" and "Hard" splits for ModelNet40 to rigorously test zero-shot performance by removing categories that were seen during pre-training.

Key Insights and Ideas

  • The "CLIP-as-Anchor" Strategy: Instead of learning a 3D-image-text space from scratch, ULIP treats the existing CLIP space as a fixed, high-quality reference. This allows the 3D encoder to learn complex semantics even with relatively small amounts of 3D data.
  • Native 3D Alignment vs. 2D Projection: Unlike previous SOTA methods like PointCLIP, which convert 3D to 2D depth maps, ULIP trains the 3D encoder to understand the point cloud natively while aligning it to the semantic space.
  • Data Efficiency: Pre-training with ULIP makes models significantly more data-efficient. A ULIP-pre-trained model often achieves better performance with only a fraction of the training data compared to a model trained from scratch.

Key Results

  • Standard Classification: ULIP improved the state-of-the-art PointMLP backbone by ~3% on the real-world ScanObjectNN dataset, reaching 89.4% Overall Accuracy.
  • Zero-Shot Performance: This is where ULIP shines. On ModelNet40, ULIP achieved 60.4% top-1 accuracy, outperforming the previous baseline, PointCLIP, by a staggering 28.8%.
  • Cross-Modal Retrieval: The model demonstrated the ability to perform image-to-3D retrieval, successfully finding 3D models that match the specific style and type of an object in a 2D query image (e.g., distinguishing between a fighter jet and an airliner).

Limitations and Future Work

  • Synthetic Gap: The pre-training relies on synthetic CAD models (ShapeNet). While it performs well on real scans (ScanObjectNN), there is still a domain gap between perfect CAD models and noisy real-world sensor data.
  • Frozen Encoders: The authors found that unfreezing the CLIP encoders led to "catastrophic forgetting" due to the small size of 3D datasets. Finding ways to fine-tune all modalities together without losing general knowledge remains a challenge.
  • Scaling: Future work could involve scaling ULIP to even larger 3D datasets or exploring more complex text descriptions beyond simple category names.

Summary updated

Abstract

The recognition capabilities of current state-of-the-art 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. In its 2D counterpart, recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. Inspired by this, leveraging multimodal information for 3D modality could be promising to improve 3D understanding under the restricted data regime, but this line of research is not well studied. Therefore, we introduce ULIP to learn a unified representation of images, texts, and 3D point clouds by pre-training with object triplets from the three modalities. To overcome the shortage of training triplets, ULIP leverages a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs. Then, ULIP learns a 3D representation space aligned with the common image-text space, using a small number of automatically synthesized triplets. ULIP is agnostic to 3D backbone networks and can easily be integrated into any 3D architecture. Experiments show that ULIP effectively improves the performance of multiple recent 3D backbones by simply pre-training them on ShapeNet55 using our framework, achieving state-of-the-art performance in both standard 3D classification and zero-shot 3D classification on ModelNet40 and ScanObjectNN. ULIP also improves the performance of PointMLP by around 3% in 3D classification on ScanObjectNN, and outperforms PointCLIP by 28.8% on top-1 accuracy for zero-shot 3D classification on ModelNet40. Our code and pre-trained models are released at https://github.com/salesforce/ULIP.
BibTeX
@inproceedings{xue2023ulip,
  author = {Le Xue and Mingfei Gao and Chen Xing and Roberto Martín-Martín and Jiajun Wu and Caiming Xiong and Ran Xu and Juan Carlos Niebles and Silvio Savarese},
  title = {{ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00120},
}
Figure