TL;DR
Segment then Splat flips the traditional 3D open-vocabulary segmentation paradigm on its head by grouping 3D Gaussians into distinct object sets before scene reconstruction begins. This simple shift eliminates geometric and semantic ambiguities, enables seamless open-vocabulary querying across both static and dynamic scenes, and delivers up to a 10x speedup in training efficiency by removing the need for an auxiliary language feature field.
What Problem Does This Paper Address?
Open-vocabulary querying in 3D space allows robots and autonomous systems to locate and interact with objects using free-form natural language (e.g., "find the chopsticks") rather than predefined categories. 3D Gaussian Splatting (3DGS) has become the go-to representation for these tasks due to its speed and rendering quality.
However, most existing methods follow a "splat then segment" approach: they reconstruct the entire scene first, often training a parallel language feature field (like CLIP) alongside the geometric reconstruction. This introduces several major flaws:
- Boundary Ambiguity: A single Gaussian can end up encoding geometric and semantic information from multiple adjacent objects, causing blurry boundaries and multi-view inconsistencies.
- Dynamic Scene Failure: In moving environments, Gaussians can suffer from "Gaussian-object misalignment," where a single Gaussian represents one object at time step $t$ and a completely different object (or background) at time step $t+1$.
- High Computational Overhead: Rendering high-dimensional language feature fields is incredibly slow and memory-intensive.
Methods
The authors introduce Segment then Splat, a framework that establishes a strict, explicit correspondence between individual 3D Gaussians and specific objects from the very beginning.
The pipeline consists of several key stages:
- Robust Object Tracking: Multi-view 2D masks are extracted across the video sequence using Segment Anything 2 (SAM2). To make this tracking robust, the authors add post-processing strategies to dynamically detect new objects, resolve pixel-level mask conflicts using an Intersection-over-Union (IoU) filter, and recover lost tracks.
- Object-Specific Gaussian Initialization: Initial sparse point clouds (from Structure-from-Motion) are assigned unique object IDs based on their visibility within the tracked 2D masks. A custom "geometric-appearances distance" metric is used to merge duplicate object tracks.
- Optimization & Reconstruction: During training, an object-level loss ($L_{obj}$) ensures that each Gaussian set contributes only to its assigned object. Crucially, all Gaussian densification and cloning operations are contained strictly within their respective object sets. For dynamic scenes, a deformation field (MLP) is applied directly to these tracked object sets.
- Multi-Granularity Scheduling: The framework tracks objects at three scales: small, middle, and large. The optimization follows a fine-to-coarse schedule (optimizing small objects first) to prevent large-scale geometry updates from destroying the internal structures of smaller sub-objects.
- CLIP Embedding Association: Instead of rendering a massive 2D language feature map, the method directly computes and assigns a single, unified CLIP embedding to each 3D object group by averaging the CLIP embeddings of its cropped 2D masks.
Datasets
The authors evaluated their approach on both static and dynamic benchmarks:
- Static Scenes: The 3DOVS dataset and the LERF_OVS dataset.
- Dynamic Scenes: The HyperNeRF dataset and the Neu3D dataset (plus additional testing on the Plenoptic Video dataset in the appendix).
Key Insights and Ideas
- Reversing the Paradigm Works: By segmenting before reconstructing, the geometric representation naturally aligns with semantic boundaries. A Gaussian never has to serve two masters (objects), yielding much cleaner 3D geometry.
- Dynamic Scenes for Free: Because the Gaussian-to-object relationship is locked in at initialization, scene dynamics (motion and deformation) can be modeled per-object without the risk of semantic drifting or misalignment over time.
- Language Fields are Overkill: Bypassing the pixel-level language field distillation entirely and instead assigning a single CLIP vector to a discrete 3D object cluster drastically slashes computational complexity without sacrificing open-vocabulary accuracy.
Key Results
Segment then Splat achieves state-of-the-art performance in both segmentation accuracy (mIoU) and training efficiency.
- Static Scenes (3DOVS): In 3D segmentation tasks, the method achieved a massive 88.53% mIoU, outperforming the closest 3D baseline (LangSplat adapted for 3D) at 47.31% and OpenGaussian at 31.00%. Remarkably, it completed training in just 9.4 minutes, compared to the ~60-70 minutes required by baselines.
- Dynamic Scenes (HyperNeRF): The method achieved 69.48% mIoU, vastly outperforming the dynamic 3D baseline, Dynamic 3D Gaussians Distillation (DGD), which scored only 7.83% due to severe misalignment issues.
- Speedup: In dynamic scenarios, Segment then Splat cut training time down from 1,564.5 minutes (DGD) to just 218 minutes—a near 10x speedup.
Additionally, because it operates natively in 3D, the model successfully retrieves the entirely occluded back-sides of objects that 2D pixel-based methods miss.
Limitations and Future Work
- Dependency on Front-End Tracking: The quality of the 3D segmentation is fundamentally bounded by the initial tracking accuracy of SAM2. In highly crowded scenes with visually identical objects, tracking failures can propagate into the reconstruction phase.
- Lack of Relational Understanding: Because CLIP embeddings are assigned to isolated object clusters, the system struggles with complex, multi-object relational queries (e.g., "the book on top of the brown table"). The authors note that incorporating inter-object contextual relationships is a vital next step.
Summary updated