Introduction
3D instance segmentation—the task of identifying and delineating individual objects within a 3D scene—is a cornerstone of spatial AI and robotics. A popular contemporary approach is "2D-to-3D lifting," which leverages high-quality masks from 2D foundation models (like Segment Anything Model, or SAM) and projects them into 3D space. However, this process is inherently error-prone. Ambiguous 2D masks often cause separate objects to be merged in 3D, and "naive" projections frequently fail to handle occlusions, causing foreground object masks to "bleed" onto background surfaces. Furthermore, many existing methods rely heavily on ground-truth depth maps, which are often unavailable or noisy in outdoor environments.
The SGS-3D Framework
SGS-3D introduces a training-free framework that treats 3D segmentation as a geometry-aware refinement process rather than a simple projection. The pipeline utilizes a novel "split-then-grow" strategy to purify noisy 2D projections and intelligently expand them into high-fidelity 3D objects.
1. Occlusion-Aware Point-Image Mapping
To eliminate reliance on sensor-provided depth, SGS-3D uses a Z-buffering technique to rasterize the 3D point cloud into a depth buffer. This allows the system to determine which 3D points are actually visible from a specific 2D camera view, effectively preventing the "bleeding" errors common in naive projections.
2. Co-occurrence Mask Filtering (CMF)
To handle noisy 2D predictions, the framework implements a consistency check across multiple views. If a 2D mask does not consistently align with the same 3D geometric primitives (superpoints) across different angles, it is pruned as unreliable.
3. The "Split-then-Grow" Strategy
This core component addresses the issue of "over-merged" masks (e.g., two adjacent chairs identified as one):
- Spatial Continuity Splitting (SCP): Lifted 3D masks are split using HDBSCAN based on spatial distance, creating "pure" semantic seeds.
- Feature-Guided Growing (FGG): These seeds are then expanded into complete objects by merging with neighboring points. This growth is guided by spatial overlap and high-level features from pre-trained models like DINOv2, ensuring fragmented parts of the same object are correctly reunited.
Experimental Evaluation
SGS-3D was evaluated on three major benchmarks representing diverse environments: ScanNet200 (indoor, 200 categories), ScanNet++ (high-fidelity indoor), and KITTI-360 (challenging outdoor scenes with complex occlusions).
Key Results
- Outdoor Superiority: On the KITTI-360 dataset, SGS-3D outperformed the previous state-of-the-art (SAI3D) by a significant 16.4% mAP, demonstrating exceptional robustness in outdoor, depth-less environments.
- Indoor Performance: The framework achieved a new performance ceiling on ScanNet200 (34.3% mAP), outperforming both training-free and several training-dependent methods.
- Efficiency and Robustness: Compared to Open3DIS, SGS-3D achieved higher accuracy while using only 2.5% of the input images and running nearly 4x faster. It also remained stable even when 50% of foreground masks were simulated as occluded.
Conclusion
SGS-3D demonstrates that 3D spatial continuity can serve as a powerful "sanity check" for 2D semantic errors. By generating its own depth buffers and employing a conservative "split first, merge later" philosophy, the framework proves that high-fidelity 3D segmentation can be achieved without fragile depth sensors or extensive training. While future work may focus on optimizing the multi-stage pipeline for real-time performance or dynamic scenes, SGS-3D currently sets a new standard for robust, training-free 3D instance segmentation.
Summary updated