TL;DR
SAMPro3D is a training-free framework that brings the power of the Segment Anything Model (SAM) to 3D instance segmentation. By anchoring prompts in 3D space and projecting them into 2D views, it ensures consistent object masks across different camera angles, often producing results that are more detailed than manual human annotations.
What Problem Does This Paper Address?
Instance segmentation in 3D—identifying and outlining every individual object in a scene—is a cornerstone of robotics and AR. Traditionally, this requires massive, laboriously labeled 3D datasets and expensive model training. While 2D foundation models like SAM have revolutionized image segmentation, "lifting" them into 3D is tricky.
Previous attempts usually fell into two traps:
- Inconsistency: Generating prompts independently for every 2D frame leads to conflicting masks that don't align when fused back into 3D.
- Emergence Issues: Propagating prompts from an initial frame to subsequent ones fails when new objects appear as the camera moves through a room.
SAMPro3D solves these by flipping the script: instead of starting with 2D prompts, it starts with prompts in 3D.
Methods
The SAMPro3D pipeline operates in a bottom-up fashion without requiring any domain-specific training:
- 3D Prompt Proposal: The system samples a set of points from the 3D point cloud (using Furthest Point Sampling) to serve as potential object "anchors."
- 3D-2D Projection: These 3D points are projected onto all available 2D RGB-D frames. Because the prompts originate from the same 3D point, the resulting pixel prompts are automatically aligned across all views.
- View-Guided Prompt Selection: Not every sampled point is a good prompt. The authors developed an algorithm to evaluate the quality of SAM-generated masks across all views, retaining only the most reliable and representative prompts for each instance.
- Surface-Based Prompt Consolidation: Large objects (like a floor or a long wall) might be segmented into pieces because a single 2D view can't see the whole thing. This step checks for intersections in 3D masked surfaces and merges prompts that are likely segmenting different parts of the same object.
- Final 3D Segmentation: All points in the scene are projected onto the segmented 2D frames, and a majority-voting scheme assigns the final 3D instance IDs.
Datasets
The researchers evaluated SAMPro3D on a wide array of benchmarks:
- Indoor Scenes: ScanNet (v2), ScanNet200, and ScanNet++.
- Outdoor Scenes: KITTI-360.
- Large View Changes: Matterport3D.
Crucially, the authors introduced ScanNet200-Fine50. They realized that SAM often segments fine-grained details (like individual cushions on a sofa) that standard datasets group into a single "sofa" label. This new benchmark provides ultra-detailed annotations for 50 scenes to properly reward zero-shot models for their precision.
Key Insights and Ideas
- 3D is the Natural Anchor: The "aha moment" is that 3D space provides a global coordinate system that 2D frames lack. By placing prompts in 3D, you solve the multi-view correspondence problem for free.
- Zero-Shot > Human Labels: In many cases, SAMPro3D's "errors" were actually instances where it was more accurate than the human annotators, who often overlook small objects or simplify complex shapes.
- Consolidation is Key: Relying solely on 2D masks is insufficient for large-scale 3D objects. Explicitly using 3D surface geometry to merge masks (Consolidation) is vital for a comprehensive scene understanding.
Key Results
SAMPro3D consistently outperformed other zero-shot and even some training-dependent methods:
- ScanNet++: It achieved a 4.5 point lead in $AP_{50}$ over the recent SAI3D and an 8.4 point lead over the training-dependent SAM-graph.
- Fine-Grained Performance: On the new ScanNet200-Fine50 dataset, SAMPro3D excelled at segmenting "tiny" and "small" objects, outperforming the state-of-the-art supervised model, Mask3D, in those categories.
- User Study: In a subjective study, 100 participants ranked SAMPro3D's results higher than both competing models and the original ScanNet200 human annotations in terms of accuracy and diversity.
Limitations and Future Work
While powerful, SAMPro3D is ultimately a wrapper for SAM. If SAM fails to segment an object in 2D (due to lighting, blur, or extreme occlusion), SAMPro3D cannot recover it in 3D. Additionally, the current version is class-agnostic; it tells you where an object is, but not what it is (e.g., "chair" vs. "table").
The authors suggest that future work could involve:
- Integrating interactive prompts to allow users to explore different levels of detail (e.g., segmenting a whole chair vs. just its legs).
- Applying the framework to NeRFs or 3D Gaussian Splatting to segment scenes directly from neural representations.
Summary updated