ClusterSplat: Semantic Cluster Selection for 3D Visual Grounding in Gaussian Splatting

Jaesung Lee, Hyeontaek Hwang, Gabriel Manalu, Daeyoung KimarXiv 2026 · arXiv

TL;DR

ClusterSplat redefines 3D visual grounding in Gaussian Splatting by shifting from noisy, per-primitive score filtering to a discrete selection process over scene-adaptive object clusters. By grouping Gaussians into coherent candidates using a description-length criterion, the model achieves state-of-the-art accuracy on ScanRefer and Ref-LERF while running over 6x faster during online inference than previous methods.

What Problem Does This Paper Address?

3D Gaussian Splatting (3DGS) has revolutionized real-time neural rendering, but "grounding"—the ability to link a natural language description (e.g., "the office chair underneath the blue painting") to specific 3D primitives—remains a challenge.

Existing methods typically assign a "relevance score" to every individual Gaussian primitive. To extract an object, they apply a heuristic threshold (like picking the top 5% of scoring primitives). However, these scores are notoriously unstable; the optimal threshold changes depending on the scene's clutter, the object's scale, and the specific wording of the prompt. As shown in the figure below, this often results in "scattered" targets where the model picks up bits of the floor or nearby furniture instead of a single, coherent object.

Figure 1: Primitive-level filtering versus cluster-level selection. Given a referring expression, primitive score fields can be unstable, and top-ratio filtering may scatter the 3D support across the target and nearby distractors. ClusterSplat instead performs scene-adaptive clustering followed by query-conditioned cluster scoring to select one coherent 3D target.

Methods

ClusterSplat introduces a two-stage framework that treats visual grounding as a discrete selection problem rather than a filtering task.

1. Offline Scene Preparation

  • Instance-Aware Feature Learning: The model first learns high-dimensional features for each Gaussian, supervised by multi-view 2D instance masks (e.g., from SAM). This ensures that primitives belonging to the same object have similar feature vectors.
  • Scene-Adaptive Clustering: Primitives are initially grouped into "seed voxels." These voxels are then merged into larger clusters using a Minimum Description Length (MDL) criterion. This allows the number and size of clusters to adapt naturally to the scene's geometry—a large bed becomes one cluster, while a small trash can becomes another—without needing a human to specify the number of objects.

2. Online Referring Expression Grounding

  • Query-Conditioned Scoring: When a user provides a text prompt, it is encoded via a frozen CLIP text encoder. A Transformer-decoder architecture then performs self-attention (to understand the scene layout) and cross-attention (to align cluster features with the text tokens).
  • Discrete Selection: Instead of thresholding, a lightweight MLP ranks the clusters. The top-ranked cluster is selected as the explicit 3D target. This selected cluster's score is then used to render a clean 2D segmentation mask.
Figure 2: ClusterSplat overview. The pipeline first performs Instance-Aware Gaussian Feature Learning and Scene-Adaptive Clustering: it learns instance-aware Gaussian features, forms feature- aware seed voxels, and merges them into scene-adaptive cluster candidates. It then performs Query- Conditioned Cluster Scoring: the referring expression conditions the cluster tokens, and the MLP scorer ranks the candidates to select the Top-1 cluster as the explicit 3D target. Finally, Cluster-Level Mask Rendering assigns the selected cluster scores to Gaussians and renders a 2D mask supervised by the mask loss.

Datasets

The authors evaluated ClusterSplat on three primary benchmarks:

  • ScanRefer: A large-scale dataset of indoor scenes with complex, natural language referring expressions.
  • ScanNet v2: Used for class-name grounding (e.g., "chair," "table") to test category-level understanding.
  • Ref-LERF: A benchmark for fine-grained spatial reasoning in smaller, highly detailed scenes.

Key Insights and Ideas

  • Granularity Matters: The "aha moment" is that individual Gaussians are too fine-grained to be the unit of semantic decision-making. By moving the decision to the cluster level, the model avoids the "hyperparameter hell" of tuning per-query thresholds.
  • MDL for Adaptivity: Using a description-length merge criterion (Eq. 2 in the paper) allows the model to balance feature consistency with the number of clusters. This prevents over-segmenting large objects while keeping small, distinct objects separate.
  • Efficiency through Discretization: Because the model only needs to rank a few dozen or hundred clusters rather than millions of primitives, the online targeting phase is extremely fast (under 20ms).

Key Results

ClusterSplat significantly outperforms prior 3DGS grounding methods like LangSplat and ReferSplat:

  • ScanRefer Accuracy: On 3D metrics (mIoU), ClusterSplat achieved 23.33%, a massive jump from ReferSplat’s 4.05%. This highlights how much more stable cluster selection is compared to primitive filtering.
  • 2D Segmentation: It reached 39.72% mIoU for rendered masks, setting a new state-of-the-art for 3DGS-based grounding.
  • Speed: Online targeting takes only 19.55 ms per query on an RTX A6000, which is roughly 6.1x faster than ReferSplat.

Limitations and Future Work

  • Single-Target Assumption: The current formulation is optimized for single-target queries. Grounding plural expressions (e.g., "the two blue chairs") would require the model to predict a variable number of clusters, which remains future work.
  • Candidate Pool Dependency: The model's performance is capped by the quality of the initial clustering. If two nearby similar objects are over-merged during the offline stage, the model cannot separate them during the online stage.
  • Offline Preparation: While the online phase is fast, the offline preparation (feature learning and clustering) takes about 125 minutes per scene, which may be a hurdle for rapid deployment in new environments.

Summary updated

Abstract

We study 3D visual grounding in 3D Gaussian Splatting (3DGS), where a referring expression should identify a target object as both a rendered 2D mask and a set of explicit 3D Gaussians. Prior methods typically extract 3D targets by applying heuristic thresholding or top-ratio filtering to dense primitive-level relevance scores. However, these scores often vary substantially across scenes and prompts, making such heuristics sensitive and leading to unstable targeting. ClusterSplat addresses this by reframing 3D target selection as cluster-level selection over scene-adaptive candidates. The method first learns instance-aware Gaussian features, forms feature-aware seed voxels, and merges neighboring seed voxels when a description-length criterion decreases, producing a scene-specific set of cluster candidates. Given a referring expression, a query-conditioned cluster scorer ranks the cluster candidates with a lightweight MLP, and the selected cluster directly defines the explicit 3D target while its cluster score is rendered for the 2D mask. On ScanRefer, ClusterSplat achieves the best rendered 2D segmentation and explicit 3D target metrics among the compared 3DGS baselines. It also obtains the best average Ref-LERF scores, supporting the same trend under fine-grained referring descriptions. On ScanNet v2 class name queries, ClusterSplat achieves the strongest rendered 2D results and the highest 3D matching accuracy, while maintaining competitive 3D IoU. Our project page is available at https://jaesung8.github.io/clustersplat/ .
BibTeX
@misc{lee2026clustersplat,
  author = {Jaesung Lee and Hyeontaek Hwang and Gabriel Manalu and Daeyoung Kim},
  title = {{ClusterSplat: Semantic Cluster Selection for 3D Visual Grounding in Gaussian Splatting}},
  year = {2026},
  eprint = {2610.36462},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36462},
}
Figure