TL;DR
CLIP-GS is a novel framework that brings open-vocabulary semantic segmentation to 3D Gaussian Splatting (3DGS) at real-time speeds (>100 FPS). By introducing compact semantic representations and cross-view consistency constraints, it significantly outperforms previous methods in accuracy—improving mIoU by over 21% on the ScanNet dataset—while remaining robust even with sparse input data.
What Problem Does This Paper Address?
Recent advances in 3D scene modeling, like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), have mastered visual reconstruction, but "understanding" what is in the scene remains difficult. Researchers have tried to bridge this gap using CLIP (Contrastive Language-Image Pre-Training), which allows users to segment objects using natural language (e.g., "find the blue sofa").
However, existing CLIP-informed 3D methods face two major hurdles:
- Efficiency Bottlenecks: Attaching high-dimensional CLIP embeddings (often 512 dimensions) to millions of 3D Gaussians makes rendering incredibly slow and memory-intensive.
- Semantic Inconsistency: 2D CLIP features are notoriously noisy and inconsistent across different camera angles. When these inconsistent 2D labels are used to train a 3D model, the resulting 3D semantics become ambiguous and "flicker" across views.
Methods
The authors propose CLIP-GS, which introduces three key innovations to solve the speed and consistency problems:
- Semantic Attribute Compactness (SAC): Instead of storing massive CLIP vectors on every Gaussian, SAC leverages the fact that pixels within an object share the same semantics. It uses the Segment Anything Model (SAM) to group regions, averages their CLIP features, and maps them to a low-dimensional semantic index. This allows the model to use tiny 3-dimensional embeddings for rasterization, which are later projected back to CLIP space via a lightweight layer.
- 3D Coherent Regularization (3DCR): To fix view inconsistency, 3DCR works in two domains. In 2D, it uses a zero-shot tracker to associate object masks across adjacent frames and applies a majority-voting strategy to unify labels. In 3D, it identifies the "dominant" Gaussians along rays from multiple views and forces them to have similar semantic features using a KL-divergence loss.
- Progressive Densification Regulation (PDR): To prevent an explosion in the number of Gaussians (which slows down rendering), PDR starts training at lower resolutions with stricter thresholds for adding new Gaussians, gradually scaling up to full resolution.
Datasets
The authors evaluated CLIP-GS on three widely used indoor scene datasets:
- ScanNet: A large-scale real-world dataset of indoor environments.
- Replica: A high-fidelity synthetic dataset with photorealistic textures.
- 3DOVS: A real-world dataset specifically designed for open-vocabulary segmentation.
They also introduced a sparse-view benchmark by sampling only 30 images per scene to test the model's robustness when data is limited.
Key Insights and Ideas
- Index-based Semantics: The shift from high-dimensional feature splatting to low-dimensional index splatting is the "secret sauce" for speed. It proves that we don't need the full CLIP vector at the primitive level to achieve high-quality results.
- The Power of Majority Voting: By treating a sequence of training images like a video and using temporal consistency (majority voting across views), the model effectively "denoises" the CLIP supervision signals.
- 3D-Aware Regularization: Simply regularizing in 2D isn't enough. By identifying the specific Gaussians that contribute most to a pixel's color across multiple views and enforcing their similarity, the model achieves much sharper object boundaries.
Key Results
- Accuracy: CLIP-GS achieved a mean Intersection over Union (mIoU) of 42.93% on ScanNet, a massive 21.20% improvement over the previous state-of-the-art (LangSplat). On the Replica dataset, it improved mIoU by 13.05%.
- Speed: The system renders at 190 FPS, significantly faster than LangSplat (45 FPS) and Feature 3DGS (2.5 FPS).
- Robustness: In sparse-view tests (only 30 images), CLIP-GS maintained high accuracy while competitors' performance degraded significantly. For example, on ScanNet sparse-view, it outperformed the next best method by 23.66% mIoU.
Limitations and Future Work
While powerful, CLIP-GS is still dependent on the quality of its 2D foundation models:
- SAM Failures: If the Segment Anything Model fails to generate a clean mask (e.g., due to motion blur), the semantic learning for those Gaussians will be imprecise.
- CLIP Limitations: In cluttered or low-contrast regions where CLIP features are weak, the model can still produce inaccurate predictions.
The authors suggest that future work could involve integrating even more robust vision-language models or developing adaptive mask refinement techniques to handle these edge cases.
Summary updated