Training-Free Hierarchical Scene Understanding for Gaussian Splatting with Superpoint Graphs

Shaohui Dai, Yansong Qu, Zheyan Li, Xinyang Li, Shengchuan Zhang, Liujuan Cao

Abstract

Recent advances in 3D reconstruction, particularly 3D Gaussian Splatting (3DGS), have enabled high-fidelity scene rendering at real-time speeds. However, adding semantic understanding to these scenes—enabling tasks like open-vocabulary object detection and part segmentation—traditionally requires computationally expensive iterative optimization. This paper presents a training-free framework that achieves hierarchical scene understanding by grouping Gaussian primitives into superpoints and utilizing a novel feature reprojection strategy. The proposed method is over 30x faster than previous state-of-the-art optimization-based approaches while delivering superior accuracy on benchmarks like LERF-OVS and 3DOVS.

Introduction

3D Gaussian Splatting has transformed the field of 3D vision by providing a fast, differentiable representation of complex scenes. Despite its success in reconstruction, making these scenes "understandable" to machines remains a significant challenge. Current methods for open-vocabulary scene understanding typically rely on lifting 2D features from foundation models (like CLIP) into 3D space through iterative backpropagation. This process is not only slow—often requiring over an hour per scene—but also prone to inconsistencies and noise across different viewpoints.

This research introduces a paradigm shift by proposing a training-free framework. By leveraging the inherent geometric structure of Gaussian scenes and the zero-shot capabilities of 2D foundation models, the system achieves high-quality semantic segmentation and hierarchical understanding in a single pass, reducing processing time from hours to seconds.

Methodology

The framework is built upon three core technical pillars designed to organize and label 3D Gaussians without the need for further training.

1. Contrastive Gaussian Partitioning

Rather than processing millions of individual Gaussian primitives, the method groups them into "superpoints." These are clusters of Gaussians that are both spatially proximal and semantically similar. The authors construct a graph where each Gaussian is a node, and edges represent spatial proximity. To prevent clusters from bleeding across object boundaries, the system incorporates 2D masks from the Segment Anything Model (SAM). These masks are used to reweight the graph edges, effectively severing connections that cross identified object boundaries in 2D views.

2. Hierarchical Superpoint Graphs

Real-world scenes are inherently hierarchical; for instance, a "car" consists of "wheels," which in turn contain "rims." To capture this, the method progressively merges superpoints into a multi-level graph. This merging is guided by SAM masks of varying granularities, allowing the system to respond to natural language queries at different scales—from entire objects to minute parts—with high precision.

3. Rendering-Guided Feature Reprojection

To populate the 3D model with semantic information, the authors use a "lifting" strategy. Instead of iterative optimization, they use a one-pass reprojection. By utilizing the transmittance (the degree to which a Gaussian contributes to a specific pixel's color), they aggregate 2D features from foundation models directly into the 3D superpoints. This weighting ensures that Gaussians most responsible for the visual appearance of an object receive the strongest semantic signals.

Experimental Results

The method was evaluated on three major benchmarks: LERF-OVS (real-world indoor scenes), 3DOVS (long-tail objects and diverse backgrounds), and ScanNet (large-scale indoor semantic segmentation).

  • Efficiency: On the LERF-OVS dataset, the framework reconstructed the semantic field in approximately 90 seconds. In comparison, LangSplat, a leading optimization-based method, required 85 minutes. This represents a speedup of 30x to 100x depending on the complexity of the scene.
  • Accuracy: Despite the lack of iterative training, the method achieved a mean Intersection-over-Union (mIoU) of 94.93% on the 3DOVS dataset, outperforming the previous state-of-the-art (93.4%).
  • Granularity: The hierarchical approach proved particularly effective at distinguishing small parts. While previous methods often produced fragmented or blurry results for small-scale queries (e.g., a "sheep's ear"), this method maintained sharp, consistent boundaries.

Key Insights

The "Low-Frequency" Nature of Semantics

A critical insight of this work is that while visual details like texture and lighting are high-frequency and change rapidly across a scene, semantic meaning is generally "low-frequency" and spatially smooth. This observation justifies the use of superpoints; optimizing high-dimensional features for every individual Gaussian is redundant when the underlying semantic label is consistent across a cluster.

2DGS vs. 3DGS

The researchers found that 2D Gaussian Splatting (2DGS), which utilizes flat disks rather than 3D ellipsoids, is superior for semantic tasks. 2DGS aligns more closely with actual object surfaces, leading to cleaner superpoint graphs and fewer "floaters" (erroneous Gaussians in empty space) that typically plague 3DGS-based semantic fields.

Conclusion and Future Work

This training-free framework demonstrates that high-quality 3D scene understanding does not require expensive iterative optimization. By combining superpoint graphs with efficient feature reprojection, the method achieves state-of-the-art results with a fraction of the temporal cost.

While highly effective, the system's performance is currently bounded by the quality of the underlying 2D foundation models (SAM and CLIP). Future research may focus on integrating more advanced graph neural networks to further refine superpoint features or adapting the framework for real-time use in robotics and autonomous navigation where rapid scene understanding is vital.

Summary updated

BibTeX
@inproceedings{dai2025training,
  author = {Shaohui Dai and Yansong Qu and Zheyan Li and Xinyang Li and Shengchuan Zhang and Liujuan Cao},
  title = {{Training-Free Hierarchical Scene Understanding for Gaussian Splatting with Superpoint Graphs}},
  booktitle = {ACM Multimedia},
  year = {2025},
  doi = {10.1145/3746027.3755146},
}
Figure