Modern 3D reconstruction frameworks, such as 3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRFs), represent environments using millions of explicit geometric primitives. A standard spatial primitive requires only a handful of parameters to describe its geometry and appearance: position, scale, rotation, opacity, and color. In total, this amounts to roughly 15 to 50 floating-point values per primitive.
However, enabling robots and autonomous systems to interact intelligently with their surroundings requires moving beyond pure geometry to embed open-vocabulary semantic understanding. To achieve this, researchers typically distill features from large Vision-Language Models (VLMs) like CLIP directly into the 3D representation.
The core challenge is dimensionality: a standard CLIP embedding spans 512 or more dimensions. Directly appending these high-dimensional vectors to millions of 3D primitives causes an unsustainable explosion in video memory (VRAM). As noted by Minghan Qin et al. in LangSplat, this storage bottleneck severely limits the scale and detail of semantic maps that can be maintained on standard hardware. Wanhua Li et al. in LangSplatV2 emphasize that this high-dimensional overhead creates a severe bottleneck, where the intensive pre-computation and memory demands of dense feature mapping offset the real-time inference benefits of explicit 3D representations.
To resolve this tension, a new class of methods focuses on compressing, decoupling, and structuring high-dimensional semantic features to make open-vocabulary 3D scene understanding scalable.
Exploiting Spatial Redundancy via Feature Grids
Why is direct storage so inefficient? The answer lies in spatial redundancy. While geometry changes rapidly at object boundaries, semantic meaning is highly continuous. Millions of distinct 3D Gaussians representing a single tabletop all share the exact same semantic concept: "table." Storing a unique 512-dimensional vector for each of these Gaussians is fundamentally redundant.
To exploit this property, recent architectures decouple the high-dimensional semantic field from the underlying geometric primitives. Instead of storing raw embeddings in each primitive, systems map 3D coordinates to a compact, low-dimensional representation that is decoded on the fly.
Naive Approach:
[Primitive] ---> [512-D CLIP Vector] (Massive VRAM footprint)
Feature Grid Approach:
[Primitive] ---> [Low-D Feature] ---> [Shared Decoder/Grid] ---> [512-D CLIP Vector]
For instance, Yuzhou Ji et al. in FastLGS: Speeding Up Language Embedded Gaussians with Feature Grid Mapping introduce a semantic feature grid that maps high-dimensional CLIP embeddings to low-dimensional features, achieving massive speedups and reducing memory overhead. Similarly, Shai Krakovsky in Lang3D-XL: Language Embedded 3D Gaussians for Large-scale Scenes utilizes a semantic bottleneck alongside a feature-based hash encoder to enable interactive query speeds on massive scenes without exhausting hardware storage.
Discrete Codebooks and Expandable Memory Banks
Another powerful intuition for reducing storage overhead is discretization. A real-world scene contains a finite, bounded number of distinct semantic categories or objects. Rather than treating the semantic space as an infinite continuum of high-dimensional vectors, we can represent it using a shared dictionary.
Under this paradigm, each 3D primitive stores a lightweight index or a low-dimensional distribution over a global set of high-dimensional vectors. Elena Alegret et al. in GALA: Guided Attention with Language Alignment for Open Vocabulary Gaussian Splatting leverage dual codebooks to align 3D primitives with natural language, drastically lowering the memory footprint while maintaining instance-consistent segmentations.
To handle open-set scenarios where new objects appear over time, Chen Liang in Open-Set Semantic Gaussian Splatting SLAM with Expandable Representation proposes an expandable feature pool. By decoupling complex semantic data from the 3D geometry into a dynamic pool, the system achieves massive memory savings suitable for consumer devices. This concept is mirrored in the work of Xueyan Zou in 3D-SPATIAL MULTIMODAL MEMORY, which replaces direct feature distillation with a compressed "memory bank" and an attention mechanism to prevent information loss during compression.
Geometry-Driven Grouping and Pre-Segmentation
A third strategy shifts the optimization target from individual primitives to geometric clusters. If we can identify which primitives belong to the same object or surface before or during the reconstruction phase, we can assign semantic labels to the entire group collectively.
Point-wise Semantics:
[Point 1 (512-D)] [Point 2 (512-D)] [Point 3 (512-D)] ...
Segment-based Semantics:
[Geometric Segment (Millions of points)] ---> Shares 1 Semantic Label
Yoshikatsu Nakajima et al. in Fast and Accurate Semantic Mapping through Geometric-based Incremental Segmentation demonstrate this by assigning semantic labels to entire geometric segments rather than individual 3D points, drastically reducing the computational and memory overhead typical of dense semantic mapping.
Taking this concept further, Yiren Lu et al. in Segment then Splat: Unified 3D Open-Vocabulary Segmentation via Gaussian Splatting invert the traditional pipeline entirely. Instead of reconstructing a scene and subsequently fitting a complex language field, the authors segment the initial 3D points into distinct object sets before reconstruction begins. This eliminates the need to learn a continuous, dense high-dimensional language field across the entire volume, accelerating training and enabling highly scalable open-vocabulary retrieval.
Key Foundational Work
SceneCode: Monocular Dense Semantic Reconstruction Using Learned Encoded Scene Representations. (2019) — contributed 1 concept elements to this direction.
Fast and Accurate Semantic Mapping through Geometric-based Incremental Segmentation (2018) — contributed 1 concept elements to this direction.