Segment Any 3D Gaussians

Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian

Abstract

Interactive 3D segmentation in radiance fields is an appealing task since its importance in 3D scene understanding and manipulation. However, existing methods face challenges in either achieving fine-grained, multi-granularity segmentation or contending with substantial computational overhead, inhibiting real-time interaction. In this paper, we introduce Segment Any 3D GAussians (SAGA), a novel 3D interactive segmentation approach that seamlessly blends a 2D segmentation foundation model with 3D Gaussian Splatting (3DGS), a recent breakthrough of radiance fields. SAGA efficiently embeds multi-granularity 2D segmentation results generated by the segmentation foundation model into 3D Gaussian point features through well-designed contrastive training. Evaluation on existing benchmarks demonstrates that SAGA can achieve competitive performance with state-of-the-art methods. Moreover, SAGA achieves multi-granularity segmentation and accommodates various prompts, including points, scribbles, and 2D masks. Notably, SAGA can finish the 3D segmentation within milliseconds, achieving nearly 1000x acceleration compared to previous SOTA.

Introduction

While 2D segmentation has been revolutionized by foundation models like the Segment Anything Model (SAM), 3D segmentation remains a high-friction task. Annotating 3D data is prohibitively expensive, and existing methods often struggle with two primary bottlenecks. First is efficiency: many methods rely on Neural Radiance Fields (NeRFs), which require heavy computation to query implicit feature fields, making real-time interaction nearly impossible. Second is multi-granularity ambiguity: in 3D space, a single point can belong to multiple entities depending on context (e.g., a wheel vs. a car).

SAGA (Segment Any 3D Gaussians) addresses these issues by leveraging the explicit nature of 3D Gaussian Splatting (3D-GS). By treating segmentation capabilities as an intrinsic attribute of the geometry itself, SAGA enables multi-granularity 3D segmentation from 2D prompts in just 4 milliseconds—orders of magnitude faster than previous radiance-field-based methods.

Methodology

The core innovation of SAGA is the Scale-Gated Affinity Feature. Instead of merely storing color and opacity, each 3D Gaussian is assigned a feature vector designed for high cosine similarity among Gaussians belonging to the same object.

Overview of the SAGA framework, illustrating the scale-gated affinity feature extraction and the scale-aware contrastive learning pipeline used to distill SAM knowledge.

To resolve hierarchical ambiguity, SAGA employs a Soft Scale-Gate Mechanism. This lightweight linear layer adjusts the magnitude of feature channels based on a specified 3D physical scale, effectively mapping the same Gaussian into different feature subspaces depending on whether the target is a small part or a large object. The model is trained via Scale-Aware Contrastive Training, distilling knowledge from SAM by taking 2D masks from various views and ensuring 3D features match 2D segmentation patterns across granularities. Finally, Local Feature Smoothing using K-Nearest Neighbors (KNN) ensures spatial consistency and mitigates noise inherent in 3D-GS.

Experimental Evaluation

SAGA was evaluated on several established benchmarks to test diverse capabilities:

  • NVOS (Neural Volumetric Object Selection): For promptable 3D segmentation.
  • SPIn-NeRF: Testing complex, real-world environments from LLFF and Mip-NeRF 360.
  • 3D-OVS: For open-vocabulary semantic segmentation.
  • Qualitative Analysis: Datasets like Tanks&Temple and Replica were used to demonstrate "segment everything" and hierarchical decomposition.

Key Results

SAGA achieves a landmark inference time of approximately 4ms. Compared to previous state-of-the-art methods like SA3D, which can take up to 45 seconds, SAGA represents a 10,000x acceleration. Despite this speed, accuracy remains high; on the NVOS dataset, SAGA achieved 92.6% mIoU, outperforming OmniSeg3D (91.7%) and SA3D-GS (92.2%).

Multi-granularity scene decomposition results. SAGA can decompose scenes into constituent parts at varying scales (S=0.01 to 1.00) almost instantly.

A standout feature is the model's ability to perform Real-Time Hierarchy decomposition. As shown in Figure 5, the model can break a scene down from a whole room into individual furniture pieces, and further into specific components (like drawers), by simply toggling the scale parameter.

Discussion and Limitations

SAGA's intelligence is inherently tied to its teacher model, SAM. If SAM fails to identify an object in 2D due to extreme scale or unusual textures, SAGA may struggle to segment it in 3D. Additionally, very small objects can occasionally be missed during automatic extraction if they are not clearly represented in multi-view 2D masks. Future work will focus on enhancing the model's generalization to recognize targets missed by the initial 2D foundation model during the distillation phase.

Summary updated

Abstract

Interactive 3D segmentation in radiance fields is an appealing task since its importance in 3D scene understanding and manipulation. However, existing methods face challenges in either achieving fine-grained, multi-granularity segmentation or contending with substantial computational overhead, inhibiting real-time interaction. In this paper, we introduce Segment Any 3D GAussians (SAGA), a novel 3D interactive segmentation approach that seamlessly blends a 2D segmentation foundation model with 3D Gaussian Splatting (3DGS), a recent breakthrough of radiance fields. SAGA efficiently embeds multi-granularity 2D segmentation results generated by the segmentation foundation model into 3D Gaussian point features through well-designed contrastive training. Evaluation on existing benchmarks demonstrates that SAGA can achieve competitive performance with state-of-the-art methods. Moreover, SAGA achieves multi-granularity segmentation and accommodates various prompts, including points, scribbles, and 2D masks. Notably, SAGA can finish the 3D segmentation within milliseconds, achieving nearly 1000x acceleration compared to previous SOTA. The project page is at https://jumpat.github.io/SAGA.
BibTeX
@misc{cen2023segment,
  author = {Jiazhong Cen and Jiemin Fang and Chen Yang and Lingxi Xie and Xiaopeng Zhang and Wei Shen and Qi Tian},
  title = {{Segment Any 3D Gaussians}},
  year = {2023},
  eprint = {2312.00860},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.00860},
}

Loading related content...

Figure