CLIP-GS: CLIP-Informed Gaussian Splatting for View-Consistent 3D Indoor Semantic Understanding

Guibiao Liao, Jiankun Li, Zhenyu Bao, Xiaoqing Ye, Qing Li, Kanglin Liu

TL;DR

CLIP-GS is a novel framework that brings open-vocabulary semantic segmentation to 3D Gaussian Splatting (3DGS) at real-time speeds (>100 FPS). By introducing compact semantic representations and cross-view consistency constraints, it significantly outperforms previous methods in accuracy—improving mIoU by over 21% on the ScanNet dataset—while remaining robust even with sparse input data.

What Problem Does This Paper Address?

Recent advances in 3D scene modeling, like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), have mastered visual reconstruction, but "understanding" what is in the scene remains difficult. Researchers have tried to bridge this gap using CLIP (Contrastive Language-Image Pre-Training), which allows users to segment objects using natural language (e.g., "find the blue sofa").

However, existing CLIP-informed 3D methods face two major hurdles:

  1. Efficiency Bottlenecks: Attaching high-dimensional CLIP embeddings (often 512 dimensions) to millions of 3D Gaussians makes rendering incredibly slow and memory-intensive.
  2. Semantic Inconsistency: 2D CLIP features are notoriously noisy and inconsistent across different camera angles. When these inconsistent 2D labels are used to train a 3D model, the resulting 3D semantics become ambiguous and "flicker" across views.
Figure 1. Visual comparisons between different CLIP-informed 3D semantic segmentation methods using the text query ”Picture” across different views. The NeRF-based method, 3DOVS [33], and 3DGS-based methods, Feature 3DGS [55] and LangSplat [37] exhibit ambigu- ous semantics and efficiency bottlenecks. In contrast, our approach achieves more precise and consistent semantic segmentation results with a faster speed.

Methods

The authors propose CLIP-GS, which introduces three key innovations to solve the speed and consistency problems:

  • Semantic Attribute Compactness (SAC): Instead of storing massive CLIP vectors on every Gaussian, SAC leverages the fact that pixels within an object share the same semantics. It uses the Segment Anything Model (SAM) to group regions, averages their CLIP features, and maps them to a low-dimensional semantic index. This allows the model to use tiny 3-dimensional embeddings for rasterization, which are later projected back to CLIP space via a lightweight layer.
  • 3D Coherent Regularization (3DCR): To fix view inconsistency, 3DCR works in two domains. In 2D, it uses a zero-shot tracker to associate object masks across adjacent frames and applies a majority-voting strategy to unify labels. In 3D, it identifies the "dominant" Gaussians along rays from multiple views and forces them to have similar semantic features using a KL-divergence loss.
  • Progressive Densification Regulation (PDR): To prevent an explosion in the number of Gaussians (which slows down rendering), PDR starts training at lower resolutions with stricter thresholds for adding new Gaussians, gradually scaling up to full resolution.
Figure 2. Illustration of CLIP-GS optimization. Left: CLIP-GS represents the 3D scene with a collection of 3D Gaussians [18] with learnable attributes, specifically adding a semantic attribute. Right: First, multi-view images undergo feature extraction using the frozen CLIP model [38] and region mask generation with SAM [20]. We then optimize CLIP-GS in an end-to-end manner through two phases. In Phase I, we introduce Semantic Attribute Compactness (SAC) to capture the unified semantics within each object, facilitating efficient optimization and rendering of semantic Gaussians. In Phase II, after training 3D Gaussians at certain iterations, we present 3D Coherent Regularization (3DCR) to enhance 3D semantic consistency. 3DCR leverages self-predicted semantics derived from CLIP-GS, refined by cross-view coherent regularization, to provide view-consistent supervision signals for optimizing Gaussians. Additionally, 3DCR identifies 3D Gaussian primitives associated with the same object through ray-based intersection matching and encourages their semantics to be similar. The color optimization process follows 3DGS [18] and is omitted for brevity.

Datasets

The authors evaluated CLIP-GS on three widely used indoor scene datasets:

  • ScanNet: A large-scale real-world dataset of indoor environments.
  • Replica: A high-fidelity synthetic dataset with photorealistic textures.
  • 3DOVS: A real-world dataset specifically designed for open-vocabulary segmentation.

They also introduced a sparse-view benchmark by sampling only 30 images per scene to test the model's robustness when data is limited.

Key Insights and Ideas

  • Index-based Semantics: The shift from high-dimensional feature splatting to low-dimensional index splatting is the "secret sauce" for speed. It proves that we don't need the full CLIP vector at the primitive level to achieve high-quality results.
  • The Power of Majority Voting: By treating a sequence of training images like a video and using temporal consistency (majority voting across views), the model effectively "denoises" the CLIP supervision signals.
  • 3D-Aware Regularization: Simply regularizing in 2D isn't enough. By identifying the specific Gaussians that contribute most to a pixel's color across multiple views and enforcing their similarity, the model achieves much sharper object boundaries.
Figure 3. Illustration of rendered segmentation maps with the text query ”Rug”. (a) and (b) correspond to training views, while (c) and (d) pertain to testing views. In (a), the rendered result appears ambiguous when immutably employing 2D CLIP semantics for Gaussian optimization. Conversely, (b) shows that leveraging 3DCR can provide coherent semantic constraints to supervise Gaussians, leading to more precise results (d). For more details, refer to Sec. 3.3.

Key Results

  • Accuracy: CLIP-GS achieved a mean Intersection over Union (mIoU) of 42.93% on ScanNet, a massive 21.20% improvement over the previous state-of-the-art (LangSplat). On the Replica dataset, it improved mIoU by 13.05%.
  • Speed: The system renders at 190 FPS, significantly faster than LangSplat (45 FPS) and Feature 3DGS (2.5 FPS).
  • Robustness: In sparse-view tests (only 30 images), CLIP-GS maintained high accuracy while competitors' performance degraded significantly. For example, on ScanNet sparse-view, it outperformed the next best method by 23.66% mIoU.

Limitations and Future Work

While powerful, CLIP-GS is still dependent on the quality of its 2D foundation models:

  • SAM Failures: If the Segment Anything Model fails to generate a clean mask (e.g., due to motion blur), the semantic learning for those Gaussians will be imprecise.
  • CLIP Limitations: In cluttered or low-contrast regions where CLIP features are weak, the model can still produce inaccurate predictions.

The authors suggest that future work could involve integrating even more robust vision-language models or developing adaptive mask refinement techniques to handle these edge cases.

Summary updated

Abstract

Exploiting 3D Gaussian Splatting (3DGS) with Contrastive Language-Image Pre-Training (CLIP) models for open-vocabulary 3D semantic understanding of indoor scenes has emerged as an attractive research focus. Existing methods typically attach high-dimensional CLIP semantic embeddings to 3D Gaussians and leverage view-inconsistent 2D CLIP semantics as Gaussian supervision, resulting in efficiency bottlenecks and deficient 3D semantic consistency. To address these challenges, we present CLIP-GS, efficiently achieving a coherent semantic understanding of 3D indoor scenes via the proposed Semantic Attribute Compactness (SAC) and 3D Coherent Regularization (3DCR). SAC approach exploits the naturally unified semantics within objects to learn compact, yet effective, semantic Gaussian representations, enabling highly efficient rendering (>100 FPS). 3DCR enforces semantic consistency in 2D and 3D domains: In 2D, 3DCR utilizes refined view-consistent semantic outcomes derived from 3DGS to establish cross-view coherence constraints; in 3D, 3DCR encourages features similar among 3D Gaussian primitives associated with the same object, leading to more precise and coherent segmentation results. Extensive experimental results demonstrate that our method remarkably suppresses existing state-of-the-art approaches, achieving mIoU improvements of 21.20% and 13.05% on ScanNet and Replica datasets, respectively, while maintaining real-time rendering speed. Furthermore, our approach exhibits superior performance even with sparse input data, substantiating its robustness.
BibTeX
@article{liao2025clip,
  author = {Guibiao Liao and Jiankun Li and Zhenyu Bao and Xiaoqing Ye and Qing Li and Kanglin Liu},
  title = {{CLIP-GS: CLIP-Informed Gaussian Splatting for View-Consistent 3D Indoor Semantic Understanding}},
  journal = {ACM Trans. Multim. Comput. Commun. Appl.},
  year = {2025},
  doi = {10.1145/3746284},
}
Figure