Introduction
3D Gaussian Splatting (3DGS) has rapidly emerged as a leading technique for high-fidelity, real-time neural rendering. However, interacting with these reconstructed 3D scenes remains a significant challenge. While existing "open-vocabulary" methods allow for segmentation based on simple category names (e.g., "chair"), they struggle with the nuance of human language. Real-world applications, such as robotics or augmented reality, require the ability to resolve referring expressions—complex natural language descriptions that involve spatial relationships and specific attributes, such as "the small cube with colored grids" or "the container with water resting on wood."
To bridge this gap, we introduce ReferSplat, an end-to-end framework designed for Referring 3D Gaussian Splatting Segmentation (R3DGS). By integrating spatial coordinates directly into the language-vision alignment process, ReferSplat enables precise object identification and segmentation, even when objects are partially occluded or viewed from challenging angles.
The Challenge of Spatial Reasoning in 3D
Existing segmentation approaches typically fall into two categories: 2D-based models and 3D open-vocabulary methods. Both have distinct limitations when applied to referring expressions:
- 2D Occlusion Sensitivity: Models like Grounded SAM operate on individual frames. If an object is occluded in a specific view, the model fails to identify it, leading to inconsistent segmentation across a 3D scene.
- Lack of Spatial Awareness: Most 3D methods treat text and visual features as abstract vectors. They often ignore the explicit 3D positional information of the points, making it difficult to resolve queries that rely on prepositions like "behind," "next to," or "to the left of."
ReferSplat addresses these issues by treating 3D coordinates as first-class features and maintaining a consistent 3D representation that leverages information across all training views.
Methodology: The ReferSplat Framework
ReferSplat transforms standard 3D Gaussians into "Referring Fields" through three primary architectural innovations:
1. 3D Gaussian Referring Fields
In addition to standard attributes like color, opacity, and covariance, each Gaussian point in ReferSplat is assigned a high-dimensional "referring feature vector." These vectors are optimized to align with text embeddings, allowing the model to compute the similarity between any 3D point and a given natural language query.
2. Position-aware Cross-Modal Interaction (PCMI)
PCMI is the core innovation of ReferSplat. Rather than relying solely on semantic features, PCMI explicitly injects 3D coordinates into the attention mechanism. The model projects 3D coordinates through a Multi-Layer Perceptron (MLP) and adds them to the semantic features. This allows the system to "ground" language in physical space. Furthermore, the mechanism infers positional context for text features based on the Gaussians they match, enabling the model to understand complex spatial relational terms.
3. Gaussian-Text Contrastive Learning (GTCL)
To distinguish between similar objects (e.g., two different red chairs), ReferSplat employs a contrastive loss strategy. In the absence of ground-truth 3D masks, the authors utilize a "top-$\\tau$ percentile" strategy. This identifies the Gaussians with the highest similarity to a text query and treats them as positive samples in the feature space, while others serve as negatives, sharpening the model's discriminative capabilities.
To facilitate training without manual 3D labels, ReferSplat uses 2D pseudo-masks generated by Grounded SAM. These masks are refined using a confidence-weighted Intersection-over-Union (IoU) strategy to ensure that only high-quality 2D labels guide the 3D learning process.
Experimental Results
Benchmarking on Ref-LERF
To evaluate the model's performance on complex queries, the authors introduced Ref-LERF, a dataset containing 295 complex language descriptions for 59 objects across four scenes. These descriptions are significantly longer and more spatially focused than previous benchmarks.
- R3DGS Performance: ReferSplat achieved 29.2 mIoU on Ref-LERF, nearly doubling the performance of the previous state-of-the-art 3D method, GS-Grouping (14.4 mIoU), and significantly outperforming 2D baselines like Grounded SAM (15.8 mIoU).
- Open-Vocabulary Tasks: On standard category-based tasks (LERF-OVS), ReferSplat reached 55.4 mIoU, demonstrating that its spatial awareness also benefits simpler classification tasks.
Efficiency and Insights
ReferSplat is designed for practical use. It trains in approximately 58 minutes (compared to nearly three hours for LangSplat) and achieves an inference speed of 26.8 FPS on an NVIDIA RTX A6000.
An interesting finding during development was that BERT embeddings outperformed CLIP for referring tasks. While CLIP is excellent at identifying nouns, BERT’s architecture is better suited for capturing the complex relational structures—such as prepositions and adjectives—essential for referring expressions.
Conclusion and Future Work
ReferSplat demonstrates that integrating explicit 3D spatial information is critical for natural language interaction with neural scenes. By moving beyond simple labels to complex referring expressions, it opens new possibilities for robotics and immersive technologies.
Future work will focus on extending this framework to dynamic environments using 4D Gaussian Splatting and scaling the Ref-LERF dataset to include more diverse and large-scale environments. Additionally, the authors aim to incorporate visual grounding to estimate precise 3D bounding boxes alongside segmentation masks.
Summary updated