SGS-3D: High-Fidelity 3D Instance Segmentation via Reliable Semantic Mask Splitting and Growing

Chaolei Wang, Yang Luo, Jing Du, Siyu Chen, Yiping Chen, Ting Han

TL;DR

SGS-3D is a training-free framework that achieves state-of-the-art 3D instance segmentation by bridging 2D foundation models and 3D geometry. Using a novel "split-then-grow" strategy, it purifies noisy 2D mask projections and intelligently expands them into high-fidelity 3D objects, even in challenging outdoor environments without relying on sensor-provided depth.

What Problem Does This Paper Address?

3D instance segmentation—the task of identifying and delineating individual objects in a 3D scene—is a cornerstone of spatial AI. Currently, many researchers use "2D-to-3D lifting," where they take high-quality masks from 2D foundation models (like SAM) and project them into 3D space.

However, this process is fraught with errors. Ambiguous 2D masks often cause separate objects to be merged in 3D, and "naive" projections often fail to handle occlusions properly (e.g., a mask for a foreground chair might "bleed" onto the wall behind it). Furthermore, many existing methods rely heavily on ground-truth depth maps from sensors, which are often noisy, incomplete, or entirely unavailable in outdoor "in-the-wild" scenarios.

Methods

The authors propose SGS-3D, a framework that treats 3D segmentation as a refinement process rather than a simple projection. The pipeline consists of three main stages:

  1. Occlusion-Aware Point-Image Mapping: Instead of relying on sensor depth, the authors use a Z-buffering technique to rasterize the 3D point cloud into a depth buffer. This allows them to determine which 3D points are actually visible from a specific 2D camera view, preventing "bleeding" errors during projection.
  2. Co-occurrence Mask Filtering (CMF): To handle noisy 2D predictions, the system checks for consistency across multiple views. If a 2D mask doesn't consistently align with the same 3D geometric primitives (superpoints) across different angles, it is pruned as unreliable.
  3. The "Split-then-Grow" Strategy:
    • Spatial Continuity Splitting (SCP): The lifted 3D masks are often "over-merged" (e.g., two adjacent chairs seen as one). SGS-3D uses HDBSCAN to split these masks based on spatial distance, creating "pure" semantic seeds.
  4. Feature-Guided Growing (FGG): These seeds are then "grown" back into complete objects by merging with neighboring points. This growth is guided by a combination of spatial overlap and high-level features from pre-trained models (like DINOv2), ensuring that fragmented parts of the same object are reunited.

Datasets

The authors evaluated SGS-3D on three major benchmarks representing diverse environments:

  • ScanNet200: A large-scale indoor dataset with 200 object categories.
  • ScanNet++: A high-fidelity indoor dataset with sub-millimeter geometry.
  • KITTI-360: A challenging outdoor dataset featuring unstructured point clouds and complex occlusions.

Key Insights and Ideas

  • Geometry as a Semantic Sanity Check: The paper’s core "aha moment" is that 3D spatial continuity is a powerful filter for 2D semantic errors. If a single 2D label projects to two spatially disconnected blobs in 3D, the geometry tells us they are likely different instances, even if they look identical to a 2D model.
  • Depth-Free Robustness: By generating their own depth buffers from the point cloud itself, the authors prove that 3D segmentation doesn't need to be tethered to fragile depth sensors. This makes the method much more viable for outdoor robotics.
  • Conservative Splitting: The philosophy of "split first, merge later" is more robust than trying to get the instance right in one shot. By breaking masks down into high-confidence "seeds," the subsequent growth phase can use more sophisticated feature matching to rebuild the object.

Key Results

  • Outdoor Superiority: On the KITTI-360 dataset, SGS-3D outperformed the previous state-of-the-art (SAI3D) by a massive 16.4% mAP, demonstrating its strength in outdoor, depth-less environments.
  • Indoor SOTA: It achieved a new performance ceiling on ScanNet200 (34.3% mAP), outperforming both training-free and several training-dependent methods.
  • Efficiency: Compared to Open3DIS, SGS-3D achieved higher accuracy while using only 2.5% of the input images and running nearly 4x faster.
  • Robustness: The system remained stable even when 50% of the foreground masks were simulated as occluded, showing high resilience to real-world data imperfections.

Limitations and Future Work

While SGS-3D is highly effective, the authors note that its multi-stage design (mapping, filtering, splitting, growing, merging) involves several sequential steps that could be further optimized for real-time performance. Additionally, the current framework is designed for static scenes; extending this "split-then-grow" logic to dynamic environments where objects move over time remains a promising but unaddressed challenge.

Summary updated

Abstract

Accurate 3D instance segmentation is crucial for high-quality scene understanding in the 3D vision domain. However, 3D instance segmentation based on 2D-to-3D lifting approaches struggle to produce precise instance-level segmentation, due to accumulated errors introduced during the lifting process from ambiguous semantic guidance and insufficient depth constraints. To tackle these challenges, we propose splitting and growing reliable semantic mask for high-fidelity 3D instance segmentation (SGS-3D), a novel "split-then-grow" framework that first purifies and splits ambiguous lifted masks using geometric primitives, and then grows them into complete instances within the scene. Unlike existing approaches that directly rely on raw lifted masks and sacrifice segmentation accuracy, SGS-3D serves as a training-free refinement method that jointly fuses semantic and geometric information, enabling effective cooperation between the two levels of representation. Specifically, for semantic guidance, we introduce a mask filtering strategy that leverages the co-occurrence of 3D geometry primitives to identify and remove ambiguous masks, thereby ensuring more reliable semantic consistency with the 3D object instances. For the geometric refinement, we construct fine-grained object instances by exploiting both spatial continuity and high-level features, particularly in the case of semantic ambiguity between distinct objects. Experimental results on ScanNet200, ScanNet++, and KITTI-360 demonstrate that SGS-3D substantially improves segmentation accuracy and robustness against inaccurate masks from pre-trained models, yielding high-fidelity object instances while maintaining strong generalization across diverse indoor and outdoor environments. Code is available at https://github.com/wangchaolei7/SGS-3D.
BibTeX
@inproceedings{wang2026sgs,
  author = {Chaolei Wang and Yang Luo and Jing Du and Siyu Chen and Yiping Chen and Ting Han},
  title = {{SGS-3D: High-Fidelity 3D Instance Segmentation via Reliable Semantic Mask Splitting and Growing}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i12.37931},
}

Loading related content...

Figure