Open-Vocabulary 3D Semantic Segmentation with Foundation Models

Li Jiang, Shaoshuai Shi, Bernt Schiele

Abstract

Open-vocabulary 3D semantic segmentation aims to recognize arbitrary object categories in 3D scenes, a task traditionally hindered by the scarcity of labeled 3D-text datasets. We present OV3D, a framework that leverages 2D images as a bridge to transfer the extensive knowledge of vision-language foundation models (e.g., CLIP, LLaVA) to 3D point clouds. By generating fine-grained, context-enriched text descriptions for 3D points, OV3D achieves state-of-the-art performance in annotation-free settings across multiple benchmarks, including ScanNet and Matterport3D.

Introduction

Traditional 3D semantic segmentation models operate under a "closed-set" assumption, limited to categories predefined during training. This restriction is impractical for real-world applications like autonomous driving or robotics, where environments are dynamic and contain an infinite variety of objects. While 2D open-vocabulary models benefit from vast internet-scale image-text data, 3D data is significantly harder to collect and label.

Conceptual overview of OV3D using 2D images as a bridge between 3D point clouds and foundation models.

OV3D addresses this data scarcity by using multi-view 2D images as an intermediary. Unlike previous methods that rely on coarse image-level captions, OV3D establishes fine-grained correspondences between 3D points and rich, context-aware textual descriptions generated by Large Vision-Language Models (LVLMs).

Methodology

The OV3D pipeline consists of three primary stages designed to create a high-fidelity mapping between 3D points and language.

Detailed pipeline of OV3D, illustrating the mapping from images to entity text, pixel association, and final point-text alignment.

1. Mapping Image to EntityText

Instead of generating a single caption for an entire image, OV3D utilizes an LVLM (specifically LLaVA-1.5) in a "chain-of-thought" conversation mode. The model is prompted to identify concrete entities and provide diverse, context-enriched descriptions. For instance, a "couch" might be described as a "blue couch, cozy seating area, or comfortable sofa." This approach captures attributes and functions, providing a more robust feature set for the 3D model.

2. Associating Pixel with EntityText

To ensure precise spatial boundaries, a Vision Foundation Model (VFM) such as SAM or SEEM is employed to generate class-agnostic segments within the 2D images. These visual segments are then matched to the LVLM-generated text descriptions using a vision-language model (CLIP), effectively labeling pixels with rich textual entities.

3. Connecting Point and EntityText via Pixel

Using geometric projection, 3D points from the point cloud are mapped onto the 2D image planes. Points falling within a labeled segment inherit the corresponding entity's text descriptions. Finally, a 3D neural network (SparseConvNet) is trained to align its point features with the CLIP text embeddings of these descriptions. At inference, the model can segment 3D clouds based on arbitrary text queries without needing 2D images.

Experiments and Results

OV3D was evaluated on several major 3D benchmarks, demonstrating superior performance in "annotation-free" (zero-shot) scenarios.

  • ScanNet & ScanNet200: OV3D reached 57.3% mIoU on ScanNet, outperforming the previous state-of-the-art (OpenScene) by 3.1%. On ScanNet200, which features a long-tail distribution of 200 classes, OV3D showed significant improvements in recognizing rare objects.
  • Matterport3D: The model achieved 45.8% mIoU, a 3.2% improvement over existing methods.
  • Cross-Domain Transfer: Testing on S3DIS after training on ScanNet confirmed the model's robust generalization capabilities.

Key Insights

  • The Power of Context: Incorporating object attributes and functions (e.g., learning that a "pool table" is for "playing billiards") allows the model to learn more discriminative features than simple category labels.
  • Fine-Grained Correspondence: By using class-agnostic segmentation as a prior, OV3D avoids the "feature smearing" common in previous methods, ensuring text descriptions are strictly tied to correct 3D points.
  • Efficiency: OV3D is significantly more storage-efficient than methods requiring the storage of massive intermediate 2D features, making it more practical for large-scale training.

Conclusion

OV3D provides a scalable and efficient framework for open-vocabulary 3D semantic segmentation. By bridging the gap between 3D geometry and 2D-based foundation models through fine-grained text descriptions, it enables robust scene understanding without manual 3D annotations. Future improvements in foundation models are expected to directly enhance OV3D's performance and versatility.

Summary updated

Abstract

In dynamic 3D environments, the ability to recognize a diverse range of objects without the constraints of predefined categories is indispensable for real-world applications. In response to this need, we introduce OV3D, an innovative framework designed for open-vocabulary 3D semantic segmentation. OV3D leverages the broad open-world knowledge embedded in vision and language foundation models to establish a fine-grained correspondence between 3D points and textual entity descriptions. These entity descriptions are enriched with contextual information, enabling a more open and comprehensive understanding. By seamlessly aligning 3D point features with entity text features, OV3D empowers open-vocabulary recognition in the 3D domain, achieving state-of-the-art open-vocabulary semantic segmentation performance across multiple datasets, including ScanNet, Matterport3D, and nuScenes.
BibTeX
@inproceedings{jiang2024open,
  author = {Li Jiang and Shaoshuai Shi and Bernt Schiele},
  title = {{Open-Vocabulary 3D Semantic Segmentation with Foundation Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02011},
}
Figure