Lang3D-XL: Language Embedded 3D Gaussians for Large-scale Scenes

Shai Krakovsky, Gal Fiebelman, Sagie Benaim, Hadar Averbuch-Elor

Abstract

Lang3D-XL introduces a highly efficient framework for embedding open-vocabulary language features into large-scale, "in-the-wild" 3D Gaussian Splatting representations. By utilizing a novel semantic bottleneck and a feature-based hash encoder, it achieves interactive query speeds (<0.1s) on massive scenes—orders of magnitude faster than previous state-of-the-art methods—while maintaining high localization accuracy for complex architectural elements.

Introduction

Creating searchable 3D worlds from unorganized Internet photo collections is a significant challenge in computer vision. While 3D Gaussian Splatting (3DGS) can reconstruct the geometry of landmarks like the Milano Cathedral, "teaching" the model to recognize specific architectural components like "spires" or "lintels" typically requires distilling features from vision-language models like CLIP.

Existing approaches face major hurdles: memory and runtime overhead for high-dimensional features, semantic misalignment in image pyramids, and the complexity of "in-the-wild" data characterized by varying lighting and transient objects. Lang3D-XL addresses these by reimagining how language features are stored and supervised.

Overview of the Lang3D-XL pipeline, illustrating the process from unorganized 'in-the-wild' images to an interactive, text-searchable 3D representation.

Methodology

The core of Lang3D-XL lies in its efficient feature representation and robust supervision strategy, designed to scale to massive environments.

Semantic Bottleneck and Feature Hashing

Instead of storing a full 512D CLIP vector per Gaussian, Lang3D-XL utilizes an extremely low-dimensional (3D) "bottleneck" feature. This feature serves as a coordinate for a Multi-Resolution Hash Encoder. Unlike standard spatial hashing, which uses XYZ coordinates, this method hashes based on feature similarity. This allows repetitive architectural structures to share representations efficiently, significantly reducing memory requirements and improving consistency across the scene.

Attenuated Downsampler and Physical Scales

To combat the "leaky" semantics of 2D CLIP pyramids—where signals for small objects often bleed into their surroundings—the authors introduce a learnable Attenuated Downsampler. This module adaptively weights features during downsampling to focus on semantically relevant regions. Furthermore, the model estimates the real-world physical size of pixels using COLMAP metadata, ensuring consistent semantic extraction regardless of camera distance.

Soft Consistency Regularization

A critical insight of the work is that hard-masking CLIP features often leads to misclassification of large structures. By penalizing variance within a Segment Anything Model (SAM) mask rather than forcing a hard label, Lang3D-XL maintains feature consistency without being misled by noisy 2D labels.

Experimental Evaluation

The model was primarily evaluated on HolyScenes, a dataset of large-scale religious landmarks with over 2,000 images per scene, as well as benchmarks like LERF-OVS and DL3DV-1K.

Interactive text-based 3D exploration results, demonstrating precise localization for queries like 'Spires' and 'Lintel'.

Key Results

  • Interactive Speed: Lang3D-XL achieves inference times of <0.1 seconds per prompt, enabling real-time exploration. This is a massive improvement over methods like HaLo-NeRF, which can take hours for similar queries.
  • Localization Accuracy: On HolyScenes, the model achieves a mean Average Precision (mAP) of 0.59, significantly outperforming other fast 3DGS distillation methods like FMGS (0.51) and LangSplat (0.19).
  • Memory Efficiency: The bottleneck approach reduces training memory to approximately 10GB, allowing the model to scale to larger scenes on consumer-grade GPUs.

Conclusion

Lang3D-XL demonstrates that efficient, large-scale 3D semantic understanding is possible by combining low-dimensional feature bottlenecks with physically-grounded scale pyramids and soft consistency constraints. While challenges remain in distinguishing extremely fine-grained decorative details, the framework provides a robust foundation for interactive exploration of reconstructed 3D environments.

Summary updated

BibTeX
@inproceedings{krakovsky2025lang,
  author = {Shai Krakovsky and Gal Fiebelman and Sagie Benaim and Hadar Averbuch-Elor},
  title = {{Lang3D-XL: Language Embedded 3D Gaussians for Large-scale Scenes}},
  booktitle = {SIGGRAPH Asia},
  year = {2025},
  doi = {10.1145/3757377.3763973},
}
Figure