Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation

Mohamed El Amine Boudjoghra, Angela Dai, Jean Lahoud, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan

TL;DR

Open-YOLO 3D delivers a massive 16× speedup in open-vocabulary 3D instance segmentation by replacing heavy 2D foundation models like SAM with efficient 2D object detectors. It achieves state-of-the-art accuracy on the ScanNet200 benchmark, processing entire scenes in just 22 seconds, making it a viable candidate for real-time robotics and AR applications.

What Problem Does This Paper Address?

3D instance segmentation—the task of identifying and delineating individual objects in a 3D point cloud—is a cornerstone of spatial AI. However, traditional "closed-set" models can only recognize classes they were explicitly trained on. "Open-vocabulary" (OV) methods solve this by allowing users to query scenes using arbitrary text prompts (e.g., "find the blue coffee mug").

The catch? Current SOTA open-vocabulary methods are agonizingly slow. They typically rely on "lifting" features from heavy 2D models like Segment Anything (SAM) and CLIP into 3D space. This process involves generating 2D masks for every frame in a video sequence, which can take 5 to 10 minutes per scene. This computational bottleneck makes them impractical for robots that need to make split-second decisions.

Methods

The authors introduce Open-YOLO 3D, which shifts the heavy lifting from 2D segmentation to 2D detection. The core philosophy is that if you already have a 3D mask proposal, using SAM to refine its 2D projection is redundant.

The pipeline consists of four main stages:

  1. 3D Object Proposals: A 3D network (like Mask3D) generates class-agnostic 3D masks from the point cloud.
  2. 2D Open-Vocabulary Detection: An efficient 2D detector (YOLO-World) processes RGB frames to generate bounding boxes and class labels for objects.
  3. Low Granularity (LG) Label Maps: Instead of expensive per-pixel masks, the system creates "LG label maps" for each frame. These are simple 2D arrays where the area inside a detected bounding box is filled with its predicted class ID.
  4. Multi-View Prompt Distribution (MVPDist): The 3D proposals are projected onto the 2D frames. The system uses a novel Accelerated Visibility Computation (VAcc)—a highly parallelized GPU tensor operation—to determine which frames provide the best view of each 3D mask. It then samples labels from the LG maps across these "top-k" views to build a probability distribution and assign the final text-based label.
Figure 2: Proposed open-world 3D instance segmentation pipeline. We use a 3D instance segmentation network (3D Network) for generating class-agnostic proposals. For open-vocabulary prediction, a 2D Open-Vocabulary Object Detector (2D OVOD) generates bounding boxes with class labels. These predictions are used to construct label maps for all input frames. Next, we assign the top-k label maps to each 3D proposal based on visibility. Finally, we generate a Multi-View Prompt Distribution from the 2D projections of the proposals to match a text prompt to every 3D proposal.

Datasets

The model was evaluated on two primary benchmarks:

  • ScanNet200: A large-scale indoor dataset with 200 categories, featuring a "long-tail" distribution of objects (head, common, and rare/tail classes).
  • Replica: A high-quality synthetic dataset used to test the model's generalizability in a zero-shot setting (trained on ScanNet, tested on Replica).

Key Insights and Ideas

  • SAM is Redundant for Labeling: The authors observed that the 2D projection of a 3D point cloud instance already contains the necessary shape information. Using SAM to generate a 2D mask just to get a label adds significant latency without a proportional gain in accuracy.
  • Bounding Boxes are Enough: By using 2D bounding boxes to create "Low Granularity" label maps, the model captures semantic information much faster than pixel-wise segmentation.
  • Visibility Filtering: Not all views are equal. By only considering the "top-k" most visible frames for a 3D mask, the model avoids noise from occlusions and motion blur, leading to cleaner label distributions.
  • Parallelized Visibility: The VAcc module replaces iterative point-counting with batch matrix multiplications on the GPU, removing one of the biggest bottlenecks in 3D-to-2D projection.

Key Results

Open-YOLO 3D outperforms existing methods in both speed and accuracy:

  • Speed: It achieves a ~16× speedup over the previous SOTA (Open3DIS), reducing inference time from several minutes to just 22 seconds per scene on ScanNet200.
  • Accuracy: On ScanNet200, it reached 24.7% mAP, an absolute gain of 2.3% over Open3DIS at the mAP50 threshold.
  • Zero-Shot Generalization: On the Replica dataset, it achieved 23.7% mAP, nearly doubling the performance of OpenMask3D (13.1%) while being significantly faster.
Figure 1: Open-vocabulary 3D instance segmentation with our Open-YOLO 3D. The proposed Open-YOLO 3D is capable of segmenting objects in a zero-shot manner. Here, We show the output for a ScanNet200 [38] scene with various prompts, where our model yields improved performance compared to the recent Open3DIS [34]. We show zoomed-in images of hidden predicted instances in the colored boxes. Additional results are in Figure 4 and suppl. material.

Limitations and Future Work

The primary limitation is the reliance on the 3D proposal network. Current 3D networks like Mask3D often struggle with very small objects due to the voxel resolution of the point cloud. If the 3D network doesn't "see" the object to create a mask, Open-YOLO 3D cannot label it, regardless of how good the 2D detector is.

The authors suggest that future work could involve fusing 2D-based 3D proposals (generated from fast 2D segmenters like FastSAM) with 3D-native proposals to better capture small-scale geometry while maintaining high inference speeds.

Summary updated

Abstract

Recent works on open-vocabulary 3D instance segmentation show strong promise, but at the cost of slow inference speed and high computation requirements. This high computation cost is typically due to their heavy reliance on 3D clip features, which require computationally expensive 2D foundation models like Segment Anything (SAM) and CLIP for multi-view aggregation into 3D. As a consequence, this hampers their applicability in many real-world applications that require both fast and accurate predictions. To this end, we propose a fast yet accurate open-vocabulary 3D instance segmentation approach, named Open-YOLO 3D, that effectively leverages only 2D object detection from multi-view RGB images for open-vocabulary 3D instance segmentation. We address this task by generating class-agnostic 3D masks for objects in the scene and associating them with text prompts. We observe that the projection of class-agnostic 3D point cloud instances already holds instance information; thus, using SAM might only result in redundancy that unnecessarily increases the inference time. We empirically find that a better performance of matching text prompts to 3D masks can be achieved in a faster fashion with a 2D object detector. We validate our Open-YOLO 3D on two benchmarks, ScanNet200 and Replica, under two scenarios: (i) with ground truth masks, where labels are required for given object proposals, and (ii) with class-agnostic 3D proposals generated from a 3D proposal network. Our Open-YOLO 3D achieves state-of-the-art performance on both datasets while obtaining up to $\sim$16$\times$ speedup compared to the best existing method in literature. On ScanNet200 val. set, our Open-YOLO 3D achieves mean average precision (mAP) of 24.7\% while operating at 22 seconds per scene. Code and model are available at github.com/aminebdj/OpenYOLO3D.
BibTeX
@inproceedings{boudjoghra2025open,
  author = {Mohamed El Amine Boudjoghra and Angela Dai and Jean Lahoud and Hisham Cholakkal and Rao Muhammad Anwer and Salman Khan and Fahad Shahbaz Khan},
  title = {{Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=CRmiX0v16e},
}
Figure