Segment then Splat: A Unified Approach for 3D Open-Vocabulary Segmentation based on Gaussian Splatting

Yiren Lu, Yunlai Zhou, Yiran Qiao, Chaoda Song, Tuo Liang, Jing Ma, Yu Yin

Abstract

Open-vocabulary querying in 3D space is crucial for enabling more intelligent perception in applications such as robotics, autonomous systems, and augmented reality. However, most existing methods rely on 2D pixel-level parsing, leading to multi-view inconsistencies and poor 3D object retrieval. Moreover, they are limited to static scenes and struggle with dynamic scenes due to the complexities of motion modeling. In this paper, we propose Segment then Splat, a 3D-aware open vocabulary segmentation approach for both static and dynamic scenes based on Gaussian Splatting. Segment then Splat reverses the long established approach of "segmentation after reconstruction" by dividing Gaussians into distinct object sets before reconstruction. Once the reconstruction is complete, the scene is naturally segmented into individual objects, achieving true 3D segmentation. This approach not only eliminates Gaussian-object misalignment issues in dynamic scenes but also accelerates the optimization process, as it eliminates the need for learning a separate language field. After optimization, a CLIP embedding is assigned to each object to enable open-vocabulary querying. Extensive experiments on various datasets demonstrate the effectiveness of our proposed method in both static and dynamic scenarios.

Introduction

Open-vocabulary 3D segmentation enables natural language querying of 3D environments (e.g., "find the red screwdriver"), a critical capability for intelligent perception in robotics and augmented reality. Conventional state-of-the-art methods typically employ a "splat then segment" paradigm, where a 3D scene is reconstructed before language features (e.g., CLIP embeddings) are distilled into a 3D "language field." However, this sequence introduces significant challenges: geometric ambiguity where Gaussians span multiple objects, multi-view inconsistencies in 2D masks, and a breakdown of object-Gaussian correspondence in dynamic scenes.

Comparison of the traditional 'Splat then Segment' pipeline versus the proposed 'Segment then Splat' approach.

The proposed "Segment then Splat" framework reverses this workflow by partitioning 3D Gaussians into distinct object sets prior to reconstruction. This fundamental shift ensures that each Gaussian is inherently tied to a specific object identity, eliminating boundary "bleeding" and providing a robust foundation for both static and dynamic scene analysis.

Methodology

The Segment then Splat pipeline consists of four integrated stages designed to maintain object integrity throughout the reconstruction process:

  1. Robust Object Tracking: Leveraging SAM and SAM2, the system generates consistent 2D masks across video frames. To address real-world tracking failures, the authors implemented modules to handle new object appearances, resolve overlapping masks, and recover tracks after temporary occlusions.
  2. Object-Specific Gaussian Initialization: Initial point clouds (from SfM or random initialization) are assigned unique Object IDs based on the 2D masks. This partitions the scene into independent Gaussian sets from the outset.
  3. Optimization & Reconstruction: During training, each Gaussian set is constrained by a specialized object-level loss ($L_{obj}$). A key innovation is the Multi-Granularity strategy, which optimizes smaller-scale objects before larger ones to prevent structural disorganization in fine-grained components.
  4. CLIP Embedding Association: Instead of learning a dense per-pixel language field, the model assigns a single, unified CLIP embedding to each optimized 3D object group. This significantly reduces computational overhead and enables efficient open-vocabulary querying.
The complete Segment then Splat framework, detailing tracking, initialization, optimization, and CLIP association.

Experimental Results

The method was evaluated across various static (3DOVS, LERF_OVS) and dynamic (HyperNeRF, Neu3D, Plenoptic Video) benchmarks. Key findings include:

  • Superior Accuracy: On the 3DOVS dataset, the method achieved a 3D mIoU of 88.53, substantially outperforming the previous 3D-based state-of-the-art, OpenGaussian (31.00).
  • Efficiency Gains: In dynamic scenarios, the approach demonstrated a 10x speedup in optimization time; for instance, requiring only 218 minutes on the HyperNeRF dataset compared to 1564 minutes for DGD.
  • Robustness: The model maintains object identity even during complex deformations or partial occlusions, thanks to the inherent 3D consistency of the Gaussian sets.

Conclusion

By enforcing object-Gaussian correspondence before optimization, Segment then Splat achieves superior boundary definition and multi-view consistency. While the method's performance is currently bounded by the quality of initial 2D trackers and lacks explicit relational modeling between objects, it represents a significant step toward efficient, 3D-aware open-vocabulary perception in both static and dynamic environments.

Summary updated

Abstract

Open-vocabulary querying in 3D space is crucial for enabling more intelligent perception in applications such as robotics, autonomous systems, and augmented reality. However, most existing methods rely on 2D pixel-level parsing, leading to multi-view inconsistencies and poor 3D object retrieval. Moreover, they are limited to static scenes and struggle with dynamic scenes due to the complexities of motion modeling. In this paper, we propose Segment then Splat, a 3D-aware open vocabulary segmentation approach for both static and dynamic scenes based on Gaussian Splatting. Segment then Splat reverses the long established approach of "segmentation after reconstruction" by dividing Gaussians into distinct object sets before reconstruction. Once the reconstruction is complete, the scene is naturally segmented into individual objects, achieving true 3D segmentation. This approach not only eliminates Gaussian-object misalignment issues in dynamic scenes but also accelerates the optimization process, as it eliminates the need for learning a separate language field. After optimization, a CLIP embedding is assigned to each object to enable open-vocabulary querying. Extensive experiments on various datasets demonstrate the effectiveness of our proposed method in both static and dynamic scenarios.
BibTeX
@inproceedings{lu2025segment,
  author = {Yiren Lu and Yunlai Zhou and Yiran Qiao and Chaoda Song and Tuo Liang and Jing Ma and Yu Yin},
  title = {{Segment then Splat: A Unified Approach for 3D Open-Vocabulary Segmentation based on Gaussian Splatting}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=ycPVp0577R},
}
Figure