Abstract
In the field of zero-shot 3D instance segmentation, existing 2D-to-3D lifting methods typically obtain 2D segmentation across multiple RGB frames using vision foundation models, which are then projected and merged into 3D space. However, since the inference of vision foundation models on a single frame is not integrated with adjacent frames, the masks of the same object may vary across different frames, leading to a lack of view consistency in the 2D segmentation. Furthermore, current lifting methods average the 2D segmentation from multiple views during the projection into 3D space, causing low-quality masks and high-quality masks to share the same weight. These factors can lead to fragmented 3D segmentation. In this paper, we present SAM2Object, a novel zero-shot 3D instance segmentation method that effectively utilizes the Segment Anything Model 2 to segment and track objects, consolidating view consistency across frames. Our approach combines these consistent 2D masks with 3D geometric priors, improving the robustness of 3D segmentation. Additionally, we introduce mask consolidation module to filter out low-quality masks across frames, which enables more precise 2D-to-3D matching. Comprehensive evaluations on ScanNetV2, ScanNet++ and ScanNet200 demonstrate the robustness and effectiveness of SAM2Object, showcasing its ability to outperform previous methods.
Introduction
3D instance segmentation is a critical task in computer vision, requiring the identification and delineation of individual objects within a 3D environment. While supervised methods have made significant strides, they rely on expensive and labor-intensive 3D annotations. To address this, "zero-shot" lifting methods have emerged, leveraging powerful 2D vision foundation models like Meta’s Segment Anything Model (SAM) to generate 2D masks from multi-view images, which are then projected into 3D space.
However, a fundamental challenge persists: view inconsistency. When a 2D model processes images frame-by-frame without temporal context, the resulting masks for the same object often vary in shape, size, or presence across different viewpoints. This inconsistency leads to fragmented and noisy 3D reconstructions.
Methodology: SAM2Object
SAM2Object addresses the view inconsistency problem by treating multi-view image sequences as video data, utilizing the temporal memory capabilities of SAM2.
Bidirectional Tracking with SAM2
Instead of independent per-frame segmentation, SAM2Object identifies keyframes where new objects appear and employs SAM2’s memory bank to track these objects both forward and backward through the image sequence. This ensures that an object identified in one view maintains a consistent identity and mask across all subsequent and preceding views.
Mask Consolidation Module
Not all 2D masks contribute equally to a high-quality 3D reconstruction. SAM2Object introduces a Mask Consolidation Module that assigns a Mask Consistency Weight to each 2D mask based on three primary factors:
- Visibility: The proportion of the 3D object visible in the current frame.
- Distance: Views closer to the camera are weighted more heavily because their projections are more accurate.
- Purity: This metric identifies "difficult" samples where a 3D segment (superpoint) projects onto multiple conflicting 2D masks, assigning them lower weights to prevent "bleeding" between objects.
Iterative Graph Clustering
The 3D scene is partitioned into geometric "superpoints." These superpoints form the nodes of a graph, with edges representing the probability of belonging to the same instance based on the consolidated 2D masks. Hierarchical clustering is then applied to the graph to produce the final 3D instance segments.
Experimental Results
The effectiveness of SAM2Object was evaluated on three major indoor 3D benchmarks: ScanNetV2, ScanNet200, and ScanNet++.
| Dataset | Metric | SAM2Object | Previous Best (Zero-Shot) |
|---|---|---|---|
| ScanNetV2 | mAP | 34.0 | 30.8 (SAI3D) |
| ScanNet++ | mAP | 20.2 | 17.1 (SAI3D) |
Notably, on ScanNetV2, SAM2Object (34.0 mAP) even outperformed the popular supervised model, Mask3D (31.1 mAP), demonstrating the power of high-quality 2D-to-3D lifting. Furthermore, the method is highly efficient, achieving these results using only 20% of the available frames in ScanNetV2 and 5% in ScanNet++.
Discussion and Conclusion
The core insight of SAM2Object is that multi-view 3D reconstruction is fundamentally a temporal consistency problem. By leveraging a video foundation model (SAM2) and a distance-aware weighting scheme, the method significantly reduces the fragmentation and noise typical of previous zero-shot lifting approaches.
While SAM2Object sets a new state-of-the-art for zero-shot 3D instance segmentation, it remains dependent on the density of the image stream for effective tracking. Future work will explore integrating 3D geometric priors more deeply into the foundation models themselves to create a more unified, 3D-aware vision system.
Summary updated