TL;DR
ClusterSplat redefines 3D visual grounding in Gaussian Splatting by shifting from noisy, per-primitive score filtering to a discrete selection process over scene-adaptive object clusters. By grouping Gaussians into coherent candidates using a description-length criterion, the model achieves state-of-the-art accuracy on ScanRefer and Ref-LERF while running over 6x faster during online inference than previous methods.
What Problem Does This Paper Address?
3D Gaussian Splatting (3DGS) has revolutionized real-time neural rendering, but "grounding"—the ability to link a natural language description (e.g., "the office chair underneath the blue painting") to specific 3D primitives—remains a challenge.
Existing methods typically assign a "relevance score" to every individual Gaussian primitive. To extract an object, they apply a heuristic threshold (like picking the top 5% of scoring primitives). However, these scores are notoriously unstable; the optimal threshold changes depending on the scene's clutter, the object's scale, and the specific wording of the prompt. As shown in the figure below, this often results in "scattered" targets where the model picks up bits of the floor or nearby furniture instead of a single, coherent object.
Methods
ClusterSplat introduces a two-stage framework that treats visual grounding as a discrete selection problem rather than a filtering task.
1. Offline Scene Preparation
- Instance-Aware Feature Learning: The model first learns high-dimensional features for each Gaussian, supervised by multi-view 2D instance masks (e.g., from SAM). This ensures that primitives belonging to the same object have similar feature vectors.
- Scene-Adaptive Clustering: Primitives are initially grouped into "seed voxels." These voxels are then merged into larger clusters using a Minimum Description Length (MDL) criterion. This allows the number and size of clusters to adapt naturally to the scene's geometry—a large bed becomes one cluster, while a small trash can becomes another—without needing a human to specify the number of objects.
2. Online Referring Expression Grounding
- Query-Conditioned Scoring: When a user provides a text prompt, it is encoded via a frozen CLIP text encoder. A Transformer-decoder architecture then performs self-attention (to understand the scene layout) and cross-attention (to align cluster features with the text tokens).
- Discrete Selection: Instead of thresholding, a lightweight MLP ranks the clusters. The top-ranked cluster is selected as the explicit 3D target. This selected cluster's score is then used to render a clean 2D segmentation mask.
Datasets
The authors evaluated ClusterSplat on three primary benchmarks:
- ScanRefer: A large-scale dataset of indoor scenes with complex, natural language referring expressions.
- ScanNet v2: Used for class-name grounding (e.g., "chair," "table") to test category-level understanding.
- Ref-LERF: A benchmark for fine-grained spatial reasoning in smaller, highly detailed scenes.
Key Insights and Ideas
- Granularity Matters: The "aha moment" is that individual Gaussians are too fine-grained to be the unit of semantic decision-making. By moving the decision to the cluster level, the model avoids the "hyperparameter hell" of tuning per-query thresholds.
- MDL for Adaptivity: Using a description-length merge criterion (Eq. 2 in the paper) allows the model to balance feature consistency with the number of clusters. This prevents over-segmenting large objects while keeping small, distinct objects separate.
- Efficiency through Discretization: Because the model only needs to rank a few dozen or hundred clusters rather than millions of primitives, the online targeting phase is extremely fast (under 20ms).
Key Results
ClusterSplat significantly outperforms prior 3DGS grounding methods like LangSplat and ReferSplat:
- ScanRefer Accuracy: On 3D metrics (mIoU), ClusterSplat achieved 23.33%, a massive jump from ReferSplat’s 4.05%. This highlights how much more stable cluster selection is compared to primitive filtering.
- 2D Segmentation: It reached 39.72% mIoU for rendered masks, setting a new state-of-the-art for 3DGS-based grounding.
- Speed: Online targeting takes only 19.55 ms per query on an RTX A6000, which is roughly 6.1x faster than ReferSplat.
Limitations and Future Work
- Single-Target Assumption: The current formulation is optimized for single-target queries. Grounding plural expressions (e.g., "the two blue chairs") would require the model to predict a variable number of clusters, which remains future work.
- Candidate Pool Dependency: The model's performance is capped by the quality of the initial clustering. If two nearby similar objects are over-merged during the offline stage, the model cannot separate them during the online stage.
- Offline Preparation: While the online phase is fast, the offline preparation (feature learning and clustering) takes about 125 minutes per scene, which may be a hurdle for rapid deployment in new environments.
Summary updated