Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding

Wenbo Zhang, Lu Zhang, Ping Hu, Liqian Ma, Yunzhi Zhuge, Huchuan Lu

Abstract

FreeGS is an unsupervised framework designed to inject view-consistent semantics into 3D Gaussian Splatting (3DGS) without the need for 2D labels or computationally expensive mask-generation preprocessing. By establishing a bootstrapping cycle between 3D spatial clustering and 2D semantic distillation, FreeGS achieves state-of-the-art performance in 3D scene understanding tasks, such as object detection and segmentation, while significantly reducing training time.

Introduction

While 3D Gaussian Splatting (3DGS) has transformed real-time 3D reconstruction, enabling semantic understanding within these scenes remains a significant challenge. Existing methods typically distill knowledge from 2D foundation models (e.g., CLIP, SAM) into 3D Gaussians. However, these approaches suffer from heavy preprocessing requirements—often taking hours to generate 2D masks—and view inconsistency, where 2D models produce varying labels for the same object from different angles, leading to "semantic flickering."

FreeGS addresses these issues by leveraging 3D geometry to guide the semantic learning process, eliminating the dependency on 2D labels and ensuring spatial consistency.

Methodology: The IDentity-coupled Semantic Field

The core innovation of FreeGS is the IDentity-coupled Semantic Field (IDSF). This field assigns two additional properties to every Gaussian: a semantic embedding and a unique instance index. These properties are optimized through a bootstrapping process that iterates between spatial grouping and feature distillation.

Overview of the FreeGS framework showing the bootstrapping cycle between 3D clustering and 2D distillation.

Union-space 3D Clustering

FreeGS groups Gaussians using a "union space" that combines geometry (position), appearance (color), and current semantic features. By employing HDBSCAN, the model identifies coherent 3D objects in an unsupervised manner, avoiding the need for predefined categories.

Multi-level 2D Semantic Distillation

The framework performs distillation at two levels:

  1. Pixel-level: High-resolution features are extracted from 2D images using MaskCLIP and FeatUP.
  2. Instance-level: 3D instance indices are rendered into 2D to create "pseudo-masks." These masks pool features from CLIP, ensuring that the entire 3D object maintains a consistent semantic identity across views.

2D-3D Joint Contrastive Learning

A contrastive loss is used to bridge the gap between 2D pixels and 3D Gaussians. This loss "pushes" features of the same object together and "pulls" different objects apart in the feature space, effectively refining the boundaries of the 3D clusters.

Experimental Results

FreeGS was evaluated on three benchmarks: LERF-Mask (long-tail scenes), 3D-OVS (open-vocabulary segmentation), and ScanNet (indoor 3D object detection).

Efficiency and Accuracy

FreeGS demonstrates superior efficiency, requiring zero preprocessing time compared to over two hours for methods like LangSplat. On the LERF-Mask dataset, FreeGS achieved a mean IoU (mIoU) of 69.5, significantly outperforming the unsupervised LERF (37.2) and rivaling SAM-supervised methods.

Qualitative comparison of semantic segmentation results against state-of-the-art methods on the Figurines and Sofa datasets.

As shown in the qualitative comparisons, FreeGS produces cleaner object boundaries and more consistent segmentations compared to baselines like LangSplat and G-Group, which often suffer from noise or incomplete masks in complex scenes.

3D Object Detection

On ScanNet, FreeGS showed a substantial leap in performance. For instance, in "Scene0462," it achieved a 14.5 mIoU, whereas LangSplat reached only 3.3 mIoU. This improvement is attributed to the geometric regularization provided by the 3D clustering, which mitigates the impact of noisy 2D-to-3D projections.

Discussion and Key Insights

  • Geometry as a Regularizer: By aligning 2D semantics with 3D clusters, FreeGS uses the inherent consistency of 3D geometry to correct the flickering issues of 2D foundation models.
  • Practical Deployment: The lack of preprocessing and the ability to perform view-independent inference (rendering the 3D semantic field directly) make FreeGS highly practical for real-world applications.
  • Limitations: The high-dimensional semantic fields increase memory overhead. Additionally, low-texture backgrounds can occasionally lead to suboptimal segmentation due to a lack of geometric cues for clustering.

Conclusion

FreeGS presents a robust, unsupervised approach to 3D scene understanding. By bootstrapping 3D geometry with 2D semantics, it achieves high-quality, view-consistent results without the overhead of traditional 2D labeling pipelines. Future work will focus on improving alignment in low-texture areas and optimizing memory usage for large-scale scenes.

Summary updated

BibTeX
@inproceedings{zhang2025bootstraping,
  author = {Wenbo Zhang and Lu Zhang and Ping Hu and Liqian Ma and Yunzhi Zhuge and Huchuan Lu},
  title = {{Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i10.33103},
}
Figure