TL;DR
This paper introduces SAMPro3D, a novel method for zero-shot 3D instance segmentation that leverages the power of the Segment Anything Model (SAM) without any additional training. By cleverly locating SAM's 2D prompts in 3D space, SAMPro3D ensures consistent segmentation across multiple views and can even segment previously unseen object categories.
What Problem Does This Paper Address?
Instance segmentation in 3D scenes is crucial for applications like augmented reality and autonomous driving. Traditional methods often struggle with "closed-set" limitations, meaning they can only segment objects they were trained on. While recent approaches have tried to use powerful 2D foundation models like SAM for 3D tasks, they often require extensive 3D pre-training or domain-specific fine-tuning, hindering their ability to generalize to entirely new scenes. A key challenge is maintaining consistency of segmentation masks across different camera views of the same 3D scene, especially when dealing with objects that might appear or disappear between views.
Methods
SAMPro3D tackles this by proposing a unique approach:
- 3D Prompt Localization: Instead of applying SAM directly to individual 2D frames and then trying to fuse inconsistent results, SAMPro3D first identifies potential "prompts" in the 3D point cloud. These 3D prompts are then projected onto the 2D RGB-D frames. This ensures that pixel prompts derived from the same 3D point are consistent across all views.
- View-Guided Prompt Selection: To refine the initial set of prompts, SAMPro3D uses a novel algorithm that examines the quality of SAM's predicted masks across all available views. This helps select the most reliable prompts for accurate segmentation.
- Surface-Based Prompt Consolidation: Recognizing that a single prompt might only capture a part of a larger object due to limited 2D views, the method consolidates prompts that segment overlapping surfaces in 3D. This ensures a more complete segmentation of individual instances.
- Zero-Shot by Design: Crucially, SAMPro3D requires no additional training on 3D data, directly inheriting SAM's zero-shot capabilities.
Datasets
The method was evaluated on several standard 3D scene understanding datasets:
- ScanNet [13]: A widely used indoor scene dataset.
- ScanNet200 [66]: A larger version of ScanNet with more object categories.
- ScanNet++ [89]: Offers more detailed segmentation masks and high-resolution images.
- KITTI-360 [42]: An outdoor suburban scene dataset.
Additionally, the authors introduced ScanNet200-Fine50, a new test set with fine-grained annotations for 50 scenes from ScanNet200, specifically designed to evaluate performance on detailed and often unannotated instances.
Key Insights and Ideas
- 3D Prompts are Key to Consistency: The core insight is that by grounding SAM's prompts in 3D space, we can achieve consistent 2D segmentations across multiple views, a major hurdle for previous methods.
- Leveraging SAM's Zero-Shot Power Directly: The paper demonstrates that SAM's impressive zero-shot capabilities can be directly transferred to 3D segmentation without the need for complex 3D-specific training pipelines.
- Fine-Grained Segmentation is Achievable: The introduction of ScanNet200-Fine50 highlights the challenge of segmenting small, detailed objects and shows that SAMPro3D excels in this area, often surpassing human annotations.
- Multi-View Information is Crucial for Prompt Selection: The View-Guided Prompt Selection algorithm effectively uses information from all camera views to filter and retain the most informative prompts.
Key Results
- Competitive Performance: SAMPro3D achieves comparable or better performance than existing zero-shot and even some fully-supervised methods on various benchmarks.
- Superiority on Fine-Grained Instances: On the new ScanNet200-Fine50 dataset, SAMPro3D significantly outperforms other methods on "tiny" and "small" instances, demonstrating its strength in fine-grained segmentation. For example, it shows a +12.1 AP50 improvement on small objects compared to SAM3D.
- Robustness: The method shows robustness across different datasets, including indoor (ScanNet, ScanNet200, ScanNet++) and outdoor (KITTI-360) scenes, and even on Matterport3D with larger view changes.
- Efficiency: The pipeline is computationally efficient, requiring only a single GPU with approximately 8000MB of memory.
Limitations and Future Work
- Direct SAM Limitations: The method inherits limitations from SAM itself, such as not directly supporting multi-granularity segmentation in the same way as the original SAM.
- Future Integration: The authors suggest future work could involve integrating user interactive prompts for more detailed exploration and applying the method to novel view synthesis techniques like NeRF or 3DGS.
- Potential for Improvement: While the method is strong, integrating enhanced SAM variants (like HQ-SAM or Mobile-SAM) can further boost performance, as shown in their experiments.
Summary updated