Introduction
Recent advancements in 3D Gaussian Splatting (3D-GS) have revolutionized the speed and quality of static scene rendering. Extensions like LangSplat have further enhanced these capabilities by embedding CLIP features, allowing for natural language queries within 3D environments. However, real-world scenes are inherently dynamic (4D), involving objects that move, change state, and perform actions.
Traditional vision-language models like CLIP struggle with these temporal nuances because they are trained on static image-text pairs. While CLIP can identify a "cup," it often fails to distinguish between an "empty cup" and a "cup being filled." 4D LangSplat addresses this by introducing a framework for building dynamic, language-aware 3D scenes capable of handling "time-sensitive" queries through the use of Multimodal Large Language Models (MLLMs).
Methods
4D LangSplat shifts from purely vision-based features to textual supervision derived from MLLMs. The pipeline integrates three core innovations to bridge the gap between static semantics and dynamic actions:
- Dual Language Fields: The model maintains two distinct semantic fields for every Gaussian point: a time-agnostic field (using CLIP) for stable properties like object identity, and a time-varying field for dynamic states like "running" or "pouring."
- Multimodal Object-Wise Video Prompting: To generate high-quality supervision, the system uses Segment Anything (SAM) to isolate objects. It then prompts an MLLM (e.g., GPT-4o) by visually highlighting the target object with contours and background blurring. This ensures the MLLM generates detailed, temporally consistent captions for specific objects, which are then converted into embeddings via an LLM.
- Status Deformable Network: To ensure smooth semantic transitions (e.g., from a "closed" to an "open" container), this network models Gaussian point features as a linear combination of $K$ prototype states, maintaining mathematical and temporal consistency.
Key Results and Insights
The framework was evaluated on the HyperNeRF and Neu3D datasets. Because these benchmarks lacked dynamic semantic labels, the authors created a ground-truth baseline using a manual annotation pipeline.
Superior Accuracy and Nuance
On the HyperNeRF dataset, 4D LangSplat outperformed deformable CLIP baselines by 29.03% in accuracy and 28.04% in Video Intersection over Union (vIoU) for time-sensitive queries. Qualitative results demonstrate the model's ability to identify subtle transitions, such as coffee diffusing in a glass or a container being opened, where static methods fail.
Efficiency and Context
By leveraging 4D Gaussian Splatting, the system achieves a query runtime of approximately 5 FPS, making it roughly 3.5x faster than existing segmentation-based 3D grouping methods. A key insight of the work is the "contour + blur" prompting strategy, which allows MLLMs to focus on specific objects without losing the background context necessary to distinguish object motion from camera movement.
Conclusion
4D LangSplat demonstrates that MLLMs are currently more effective at describing actions in text than vision models are at producing dense features for those actions. While the method is currently limited by the representational capacity of the underlying MLLM and the reliability of initial object tracking, it sets a new standard for semantic tracking in evolving 4D environments. Future work may involve deeper integration with world models to better reason about physics and object permanence.
Summary updated