Abstract
Open-vocabulary 3D semantic segmentation aims to recognize arbitrary object categories in 3D scenes, a task traditionally hindered by the scarcity of labeled 3D-text datasets. We present OV3D, a framework that leverages 2D images as a bridge to transfer the extensive knowledge of vision-language foundation models (e.g., CLIP, LLaVA) to 3D point clouds. By generating fine-grained, context-enriched text descriptions for 3D points, OV3D achieves state-of-the-art performance in annotation-free settings across multiple benchmarks, including ScanNet and Matterport3D.
Introduction
Traditional 3D semantic segmentation models operate under a "closed-set" assumption, limited to categories predefined during training. This restriction is impractical for real-world applications like autonomous driving or robotics, where environments are dynamic and contain an infinite variety of objects. While 2D open-vocabulary models benefit from vast internet-scale image-text data, 3D data is significantly harder to collect and label.
OV3D addresses this data scarcity by using multi-view 2D images as an intermediary. Unlike previous methods that rely on coarse image-level captions, OV3D establishes fine-grained correspondences between 3D points and rich, context-aware textual descriptions generated by Large Vision-Language Models (LVLMs).
Methodology
The OV3D pipeline consists of three primary stages designed to create a high-fidelity mapping between 3D points and language.
1. Mapping Image to EntityText
Instead of generating a single caption for an entire image, OV3D utilizes an LVLM (specifically LLaVA-1.5) in a "chain-of-thought" conversation mode. The model is prompted to identify concrete entities and provide diverse, context-enriched descriptions. For instance, a "couch" might be described as a "blue couch, cozy seating area, or comfortable sofa." This approach captures attributes and functions, providing a more robust feature set for the 3D model.
2. Associating Pixel with EntityText
To ensure precise spatial boundaries, a Vision Foundation Model (VFM) such as SAM or SEEM is employed to generate class-agnostic segments within the 2D images. These visual segments are then matched to the LVLM-generated text descriptions using a vision-language model (CLIP), effectively labeling pixels with rich textual entities.
3. Connecting Point and EntityText via Pixel
Using geometric projection, 3D points from the point cloud are mapped onto the 2D image planes. Points falling within a labeled segment inherit the corresponding entity's text descriptions. Finally, a 3D neural network (SparseConvNet) is trained to align its point features with the CLIP text embeddings of these descriptions. At inference, the model can segment 3D clouds based on arbitrary text queries without needing 2D images.
Experiments and Results
OV3D was evaluated on several major 3D benchmarks, demonstrating superior performance in "annotation-free" (zero-shot) scenarios.
- ScanNet & ScanNet200: OV3D reached 57.3% mIoU on ScanNet, outperforming the previous state-of-the-art (OpenScene) by 3.1%. On ScanNet200, which features a long-tail distribution of 200 classes, OV3D showed significant improvements in recognizing rare objects.
- Matterport3D: The model achieved 45.8% mIoU, a 3.2% improvement over existing methods.
- Cross-Domain Transfer: Testing on S3DIS after training on ScanNet confirmed the model's robust generalization capabilities.
Key Insights
- The Power of Context: Incorporating object attributes and functions (e.g., learning that a "pool table" is for "playing billiards") allows the model to learn more discriminative features than simple category labels.
- Fine-Grained Correspondence: By using class-agnostic segmentation as a prior, OV3D avoids the "feature smearing" common in previous methods, ensuring text descriptions are strictly tied to correct 3D points.
- Efficiency: OV3D is significantly more storage-efficient than methods requiring the storage of massive intermediate 2D features, making it more practical for large-scale training.
Conclusion
OV3D provides a scalable and efficient framework for open-vocabulary 3D semantic segmentation. By bridging the gap between 3D geometry and 2D-based foundation models through fine-grained text descriptions, it enables robust scene understanding without manual 3D annotations. Future improvements in foundation models are expected to directly enhance OV3D's performance and versatility.
Summary updated