SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation

Mutian Xu, Xingyilang Yin, Lingteng Qiu, Yang Liu, Xin Tong, Xiaoguang Han

TL;DR

SAMPro3D is a training-free framework that brings the power of the Segment Anything Model (SAM) to 3D instance segmentation. By anchoring prompts in 3D space and projecting them into 2D views, it ensures consistent object masks across different camera angles, often producing results that are more detailed than manual human annotations.

What Problem Does This Paper Address?

Instance segmentation in 3D—identifying and outlining every individual object in a scene—is a cornerstone of robotics and AR. Traditionally, this requires massive, laboriously labeled 3D datasets and expensive model training. While 2D foundation models like SAM have revolutionized image segmentation, "lifting" them into 3D is tricky.

Previous attempts usually fell into two traps:

  1. Inconsistency: Generating prompts independently for every 2D frame leads to conflicting masks that don't align when fused back into 3D.
  2. Emergence Issues: Propagating prompts from an initial frame to subsequent ones fails when new objects appear as the camera moves through a room.

SAMPro3D solves these by flipping the script: instead of starting with 2D prompts, it starts with prompts in 3D.

Methods

The SAMPro3D pipeline operates in a bottom-up fashion without requiring any domain-specific training:

  1. 3D Prompt Proposal: The system samples a set of points from the 3D point cloud (using Furthest Point Sampling) to serve as potential object "anchors."
  2. 3D-2D Projection: These 3D points are projected onto all available 2D RGB-D frames. Because the prompts originate from the same 3D point, the resulting pixel prompts are automatically aligned across all views.
  3. View-Guided Prompt Selection: Not every sampled point is a good prompt. The authors developed an algorithm to evaluate the quality of SAM-generated masks across all views, retaining only the most reliable and representative prompts for each instance.
  4. Surface-Based Prompt Consolidation: Large objects (like a floor or a long wall) might be segmented into pieces because a single 2D view can't see the whole thing. This step checks for intersections in 3D masked surfaces and merges prompts that are likely segmenting different parts of the same object.
  5. Final 3D Segmentation: All points in the scene are projected onto the segmented 2D frames, and a majority-voting scheme assigns the final 3D instance IDs.
Figure 2. The comparison of our key idea and others. Our method (b) locates SAM prompts in 3D, which aligns pixel prompts across frames, bringing the frame consistency of prompts and their masks, and can handle newly emerged instances. Here we use random colors to visualize 3D results for instance discrimination, so there is no correlation between the colors assigned to 2D and 3D instances.

Datasets

The researchers evaluated SAMPro3D on a wide array of benchmarks:

  • Indoor Scenes: ScanNet (v2), ScanNet200, and ScanNet++.
  • Outdoor Scenes: KITTI-360.
  • Large View Changes: Matterport3D.

Crucially, the authors introduced ScanNet200-Fine50. They realized that SAM often segments fine-grained details (like individual cushions on a sofa) that standard datasets group into a single "sofa" label. This new benchmark provides ultra-detailed annotations for 50 scenes to properly reward zero-shot models for their precision.

Figure 5. Qualitative comparison of our method, SAM3D [88], Mask3D [69] and ScanNet200’s annotations [66], across various scenes in ScanNet200, from holistic to focused view. Mask3D does not treat floor and wall as instances, resulting in the absence of these two labels in its results. Better view in zoom and color.

Key Insights and Ideas

  • 3D is the Natural Anchor: The "aha moment" is that 3D space provides a global coordinate system that 2D frames lack. By placing prompts in 3D, you solve the multi-view correspondence problem for free.
  • Zero-Shot > Human Labels: In many cases, SAMPro3D's "errors" were actually instances where it was more accurate than the human annotators, who often overlook small objects or simplify complex shapes.
  • Consolidation is Key: Relying solely on 2D masks is insufficient for large-scale 3D objects. Explicitly using 3D surface geometry to merge masks (Consolidation) is vital for a comprehensive scene understanding.

Key Results

SAMPro3D consistently outperformed other zero-shot and even some training-dependent methods:

  • ScanNet++: It achieved a 4.5 point lead in $AP_{50}$ over the recent SAI3D and an 8.4 point lead over the training-dependent SAM-graph.
  • Fine-Grained Performance: On the new ScanNet200-Fine50 dataset, SAMPro3D excelled at segmenting "tiny" and "small" objects, outperforming the state-of-the-art supervised model, Mask3D, in those categories.
  • User Study: In a subjective study, 100 participants ranked SAMPro3D's results higher than both competing models and the original ScanNet200 human annotations in terms of accuracy and diversity.
Figure 5. Qualitative comparison of our method, SAM3D [88], Mask3D [69] and ScanNet200’s annotations [66], across various scenes in ScanNet200, from holistic to focused view. Mask3D does not treat floor and wall as instances, resulting in the absence of these two labels in its results. Better view in zoom and color.

Limitations and Future Work

While powerful, SAMPro3D is ultimately a wrapper for SAM. If SAM fails to segment an object in 2D (due to lighting, blur, or extreme occlusion), SAMPro3D cannot recover it in 3D. Additionally, the current version is class-agnostic; it tells you where an object is, but not what it is (e.g., "chair" vs. "table").

The authors suggest that future work could involve:

  • Integrating interactive prompts to allow users to explore different levels of detail (e.g., segmenting a whole chair vs. just its legs).
  • Applying the framework to NeRFs or 3D Gaussian Splatting to segment scenes directly from neural representations.

Summary updated

Abstract

We introduce SAMPro3D for zero-shot instance segmentation of 3D scenes. Given the 3D point cloud and multiple posed RGB-D frames of 3D scenes, our approach segments 3D instances by applying the pretrained Segment Anything Model (SAM) to 2D frames. Our key idea involves locating SAM prompts in 3D to align their projected pixel prompts across frames, ensuring the view consistency of SAM-predicted masks. Moreover, we suggest selecting prompts from the initial set guided by the information of SAM-predicted masks across all views, which enhances the overall performance. We further propose to consolidate different prompts if they are segmenting different surface parts of the same 3D instance, bringing a more comprehensive segmentation. Notably, our method does not require any additional training. Extensive experiments on diverse benchmarks show that our method achieves comparable or better performance compared to previous zero-shot or fully supervised approaches, and in many cases surpasses human annotations. Furthermore, since our fine-grained predictions often lack annotations in available datasets, we present ScanNet200-Fine50 test data which provides fine-grained annotations on 50 scenes from ScanNet200 dataset. The project page can be accessed at https://mutianxu.github.io/sampro3d/.
BibTeX
@inproceedings{xu2025sampro,
  author = {Mutian Xu and Xingyilang Yin and Lingteng Qiu and Yang Liu and Xin Tong and Xiaoguang Han},
  title = {{SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation}},
  booktitle = {3DV},
  year = {2025},
  doi = {10.1109/3dv66043.2025.00116},
}
Figure