The central challenge in 3D computer vision is a data imbalance: while we have massive datasets of labeled 2D images, high-quality 3D semantic data is notoriously difficult and expensive to collect. To bridge this gap, a new research direction has emerged that "lifts" the intelligence of 2D Vision Foundation Models (VFMs)—like the Segment Anything Model (SAM) and CLIP—into 3D representations.
By projecting 2D semantic masks and language features into 3D space, researchers are creating systems that can segment, identify, and manipulate 3D objects with zero manual 3D labels.
The Mechanism of Semantic Lifting
The core intuition behind this research is that 2D models already understand "what" an object is, while 3D reconstructions (like point clouds or NeRFs) understand "where" things are. "Lifting" is the process of fusing these two.
In frameworks like SAM3D, Xiaoyang Wu and colleagues demonstrate how 2D masks from SAM can be projected into a 3D point cloud. Because a single 3D object is viewed from multiple 2D angles, the system must merge these overlapping projections. This is often achieved through bidirectional merging or "voting" mechanisms, where 3D points accumulate semantic identities from the 2D frames that see them.
This approach allows for "zero-shot" understanding. For example, OV-SAM3D by Hanchen Tai and others combines SAM for segmentation with CLIP for labeling, enabling a robot to find a "red coffee mug" in a 3D room without ever being trained on 3D mug data.
Embedding Semantics in Neural Radiance and Splatting
While point clouds are discrete, modern 3D reconstruction often uses continuous representations like Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS). Researchers are now embedding 2D foundation features directly into these volumes.
- Semantic Scaffolds: In InstanceGaussian, Haijie Li introduces a "Semantic-Scaffold" that allows 3DGS to achieve instance-level perception. Each Gaussian "splat" carries not just color and opacity, but a semantic identity.
- Identity Encoding: Mingqiao Ye and the team behind Gaussian Grouping augment Gaussians with a learnable "Identity Encoding." During training, these encodings are supervised by 2D masks from SAM, effectively teaching the 3D model to group its components into coherent objects.
- Language Fields: Methods like CLIP-GS by Guibiao Liao distill CLIP features into the 3D representation. This enables real-time natural language queries—users can type "chair" and the model highlights the corresponding 3D Gaussians.
Resolving Multi-View Inconsistency
A recurring technical hurdle is "index inconsistency." If you segment a chair in Frame A and Frame B, an automated 2D tool might assign them different IDs (e.g., "Object 1" and "Object 5").
To solve this, researchers like Xinhua Cheng in the PCFF framework use metric learning to ensure that the same object receives a consistent feature signature across different views. Other approaches, such as GraphSeg by Haozhan Tang, treat the problem as a graph optimization, merging over-segmented 2D fragments into a unified 3D whole based on geometric proximity.
The 2D-to-3D Bottleneck
The most critical insight across these 94 papers is the existence of a fundamental performance ceiling. Because these 3D systems rely on 2D "teachers," they inherit the teachers' blind spots.
As noted in SAI3D by Yingda Yin and OpenObj by Yinan Deng, if a 2D foundation model fails to detect an object in the source images—perhaps due to poor lighting, motion blur, or unusual camera angles—the 3D system has no mechanism to "hallucinate" or recover that object. The 3D reconstruction is strictly bounded by the perception of the underlying 2D model.
Why This Matters
This direction transforms 3D scene understanding from a supervised learning problem into an aggregation problem. By leveraging the "common sense" of models like SAM and CLIP, researchers are enabling robots to navigate and interact with the world (O2V-Mapping) and allowing creators to edit 3D scenes as easily as they edit 2D photos (SSDNeRF). The future of 3D vision is not in collecting more 3D labels, but in better ways to lift the 2D knowledge we already have.