4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models

Wanhua Li, Renping Zhou, Jiawei Zhou, Yingwei Song, Johannes Herter, Minghan Qin, Gao Huang, Hanspeter Pfister

Introduction

Recent advancements in 3D Gaussian Splatting (3D-GS) have revolutionized the speed and quality of static scene rendering. Extensions like LangSplat have further enhanced these capabilities by embedding CLIP features, allowing for natural language queries within 3D environments. However, real-world scenes are inherently dynamic (4D), involving objects that move, change state, and perform actions.

Traditional vision-language models like CLIP struggle with these temporal nuances because they are trained on static image-text pairs. While CLIP can identify a "cup," it often fails to distinguish between an "empty cup" and a "cup being filled." 4D LangSplat addresses this by introducing a framework for building dynamic, language-aware 3D scenes capable of handling "time-sensitive" queries through the use of Multimodal Large Language Models (MLLMs).

Methods

4D LangSplat shifts from purely vision-based features to textual supervision derived from MLLMs. The pipeline integrates three core innovations to bridge the gap between static semantics and dynamic actions:

  1. Dual Language Fields: The model maintains two distinct semantic fields for every Gaussian point: a time-agnostic field (using CLIP) for stable properties like object identity, and a time-varying field for dynamic states like "running" or "pouring."
  2. Multimodal Object-Wise Video Prompting: To generate high-quality supervision, the system uses Segment Anything (SAM) to isolate objects. It then prompts an MLLM (e.g., GPT-4o) by visually highlighting the target object with contours and background blurring. This ensures the MLLM generates detailed, temporally consistent captions for specific objects, which are then converted into embeddings via an LLM.
  3. Status Deformable Network: To ensure smooth semantic transitions (e.g., from a "closed" to an "open" container), this network models Gaussian point features as a linear combination of $K$ prototype states, maintaining mathematical and temporal consistency.
The 4D LangSplat pipeline: from object-wise video prompting with MLLMs to the Status Deformable Network.

Key Results and Insights

The framework was evaluated on the HyperNeRF and Neu3D datasets. Because these benchmarks lacked dynamic semantic labels, the authors created a ground-truth baseline using a manual annotation pipeline.

Superior Accuracy and Nuance

On the HyperNeRF dataset, 4D LangSplat outperformed deformable CLIP baselines by 29.03% in accuracy and 28.04% in Video Intersection over Union (vIoU) for time-sensitive queries. Qualitative results demonstrate the model's ability to identify subtle transitions, such as coffee diffusing in a glass or a container being opened, where static methods fail.

Qualitative results showing RGB input vs. the learned 4D Field for coffee pouring and container opening scenes.

Efficiency and Context

By leveraging 4D Gaussian Splatting, the system achieves a query runtime of approximately 5 FPS, making it roughly 3.5x faster than existing segmentation-based 3D grouping methods. A key insight of the work is the "contour + blur" prompting strategy, which allows MLLMs to focus on specific objects without losing the background context necessary to distinguish object motion from camera movement.

Conclusion

4D LangSplat demonstrates that MLLMs are currently more effective at describing actions in text than vision models are at producing dense features for those actions. While the method is currently limited by the representational capacity of the underlying MLLM and the reliability of initial object tracking, it sets a new standard for semantic tracking in evolving 4D environments. Future work may involve deeper integration with world models to better reason about physics and object permanence.

Summary updated

BibTeX
@inproceedings{li2025d,
  author = {Wanhua Li and Renping Zhou and Jiawei Zhou and Yingwei Song and Johannes Herter and Minghan Qin and Gao Huang and Hanspeter Pfister},
  title = {{4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02049},
}
Figure