Traditional 3D scene understanding relies on closed-set paradigms. Models are trained to recognize a fixed list of categories—such as "chair," "table," or "door"—derived from painstakingly labeled 3D datasets. However, the real world presents an infinite variety of object categories, visual states, and functional descriptions. A robot operating in a home cannot rely on a fixed vocabulary; it must understand arbitrary queries like "the circular table" or "something to sit on."
To bypass the scarcity of labeled 3D data, a powerful insight has emerged: instead of training 3D models on semantic labels from scratch, we can lift the open-vocabulary knowledge of 2D vision-language foundation models directly into 3D space. By aligning 3D geometric primitives (such as voxels, point clouds, Neural Radiance Fields, or Gaussian splats) with a shared language-vision embedding space, the resulting 3D representations inherit the ability to resolve open-ended natural language queries without requiring explicit 3D semantic training.
2D Images ───► [ 2D Foundation Model (CLIP/SAM) ] ───► Language-Aligned Features
│
(Multi-View Distillation)
▼
3D Geometry ─────────────────────────────────────────► [ Open-Vocabulary 3D Map ]
The Core Mechanism: Multi-View Feature Distillation
The fundamental challenge in open-vocabulary 3D understanding is bridging the gap between flexible language descriptions and fixed 3D geometric structures, as noted by Elena Alegret et al. in their work on GALA. The mechanism that makes this bridge possible is multi-view consistency.
When a 3D scene is captured from multiple angles, any given point in 3D space projects to corresponding pixel coordinates across several 2D images. By passing these 2D images through a pre-trained vision-language model (like CLIP), we obtain dense, language-aligned feature maps. We can then optimize a 3D representation so that its internal features, when projected back into 2D, match the embeddings generated by the foundation model.
Sosuke Kobayashi et al. pioneered this approach with Distilled Feature Fields in Decomposing NeRF for Editing via Feature Field Distillation. By distilling 2D features from foundation models into a 3D volume alongside a standard radiance field, the authors enabled users to isolate and manipulate specific objects using arbitrary text queries. Because the underlying 2D features are already mapped to a text-embedding space, the 3D field naturally inherits this open-set capability.
Primitives and Fields: Continuous 3D Representations
This distillation paradigm adapts naturally to different 3D scene representations, primarily Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS).
Volumetric Fields (NeRFs)
In volumetric approaches, text-aligned embeddings are optimized within a continuous spatial field. Guibiao Liao et al. demonstrated in OV-NeRF that the inherent 3D consistency of a NeRF can actually correct errors and inconsistencies present in 2D foundation model outputs. Similarly, Hao Zhang et al. developed Open-NeRF, which aligns 2D mask proposals across multiple viewpoints before distilling them into the 3D field, ensuring that rare or long-tail objects are accurately captured and segmented.
Point Primitives (Gaussian Splatting)
While NeRFs offer high fidelity, they are computationally expensive to query and render. Recent work shifts toward embedding language features directly into explicit 3D primitives.
- Dynamic Scenes: In DGD: Dynamic 3D Gaussians Distillation, Isaac Labe et al. distilled high-dimensional features into dynamic 3D Gaussians, allowing for real-time tracking and semantic editing of moving objects from a single video.
- Collaborative Mapping: Javier Yu et al. extended this to robotics in HAMMER, creating a multi-robot framework where heterogeneous agents collaboratively construct a searchable, continuous feature field using 3D Gaussian Splatting.
- Specialized Domains: The flexibility of this approach allows it to generalize to highly technical environments. For instance, Long Bai et al. introduced SurgTPGS, mapping 3D spatial data to language embeddings in the operating room. This allows surgeons to localize specific anatomical structures and surgical tools using natural language prompts without requiring fixed label sets during training.
Object-Centric and Voxel-Based Mapping
Continuous feature fields are highly expressive, but robotic manipulation and navigation often require discrete, object-centric abstractions. Instead of embedding features into every point or voxel, several frameworks focus on isolating 3D instance proposals first, then assigning them language embeddings.
[ 3D Scene Data ] ───► Class-Agnostic Mask Proposals ───► [ 3D Instances ]
│
(Assign CLIP Embedding)
▼
Text Query ─────────► [ Cosine Similarity Match ] ──────► Selected Instance
Ayça Takmaz et al. proposed this object-centric strategy in OpenMask3D: Open-Vocabulary 3D Instance Segmentation. The framework generates class-agnostic 3D mask proposals from the geometry alone, then refines and assigns multi-view CLIP embeddings to each mask using the Segment Anything Model (SAM). This allows the system to identify objects based on abstract properties like material, geometry, or state (e.g., "circular table") rather than just categorical labels.
To optimize computational efficiency, Yesol Park et al. introduced OV-MAP. Instead of computing per-voxel embeddings, the authors identified the single most representative 2D view for each reconstructed 3D instance and extracted its semantic features using a vision-language model. This drastically reduces the memory footprint while maintaining zero-shot retrieval accuracy.
For spatial navigation, frameworks like FindThis by Arjun Majumdar et al. and Demonstrating OK-Robot by Jay Vakil et al. aggregate 2D open-vocabulary object detections and language embeddings into voxelized spatial maps. This allows a robot to perform language-based coordinate lookups in physical space, translating vague human commands into precise 3D coordinates without prior training on that specific environment.
Beyond Categorization: Affordances and Functional Reasoning
Because these lifted 3D representations share an embedding space with large language models, the queries are not restricted to nouns. The system can reason about the utility and affordance of geometric regions.
Yicong Li et al. explored this in LASO: Language-Guided Affordance Segmentation on 3D Object, replacing fixed category labels with dynamic queries derived from natural language instructions. By grounding semantic concepts like actions or tasks directly into 3D coordinates, the model can identify the specific functional region of an object (e.g., the handle of a mug when asked "how would you pick this up?").
As demonstrated by Jinlong Peng et al. in UniM-OV3D, aligning 3D data with broad linguistic concepts allows models to leverage the general world knowledge embedded in language models. The 3D scene representation ceases to be a static collection of geometric shapes and becomes an interactive, semantically rich environment capable of understanding context, state, and human intent.
Key Foundational Work
CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition (2023) — contributed 2 concept elements to this direction.
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023) — contributed 2 concept elements to this direction.
POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images (2023) — contributed 1 concept elements to this direction.