3D perception tasks—such as semantic segmentation, object detection, and instance tracking on point clouds or meshes—are notoriously bottlenecked by data scarcity. Manually annotating a 3D scene requires drawing 3D bounding boxes or assigning per-point labels to unstructured spatial data, a process that is orders of magnitude more expensive than labeling 2D images.
Meanwhile, the 2D vision domain has experienced a revolution driven by massive foundation models like the Segment Anything Model (SAM) and Contrastive Language-Image Pre-training (CLIP). These models possess robust, zero-shot, open-vocabulary capabilities trained on billions of images.
Cross-modal pseudo-label projection bridges this dimensional divide. Instead of training 3D models from scratch on scarce human annotations, this paradigm mathematically projects 3D spatial data into 2D image planes, queries mature 2D foundation models to obtain dense semantic or instance labels, and maps those labels back into 3D space. By treating 2D models as automated supervisors, researchers can generate massive 3D training sets or perform zero-shot 3D scene understanding without a single manual 3D label.
The Geometry of Knowledge Transfer
The core mechanism relies on a simple geometric truth: if you know the spatial position of a 3D point and the parameters of a camera observing it, you can deterministically calculate exactly which pixel that point maps to.
[ 3D Point Cloud / Mesh ] ---(Perspective Projection)---> [ 2D Image Plane ]
│
(Query 2D Foundation Model)
│
▼
[ 3D Pseudo-Labels ] <---(Back-Projection Head)---- [ 2D Semantic Masks ]
Early iterations of this concept established the baseline pipeline. For instance, J. Boehm in SEMANTIC SEGMENTATION OF TERRESTRIAL LIDAR DATA USING CO-REGISTERED RGB DATA demonstrated that formatting terrestrial LiDAR data as 2D perspective images allows off-the-shelf 2D convolutional networks to segment the scene, bypassing the need for native 3D training data. Similarly, Florian Piewak developed an automated pipeline in Boosting LiDAR-Based Semantic Labeling by Cross-modal Training Data Generation that projects LiDAR points into a camera's field of view to inherit labels from an image segmentation model, drastically boosting 3D model performance with minimal manual fine-tuning.
This geometric bridge works in reverse to elevate human annotations as well. For example, Fan Yan in ONCE-3DLanes: Building Monocular 3D Lane Detection generated high-fidelity 3D lane ground truths by projecting 2D human-annotated lane markers onto synchronized LiDAR point clouds, using the depth sensor to supply the missing height dimension.
Lifting 2D Foundation Models to 3D Space
With the advent of open-vocabulary 2D foundation models, this projection mechanism evolved from a simple data-generation trick into a framework for zero-shot 3D understanding. Instead of relying on closed-set 2D classifiers, modern architectures project open-vocabulary features directly onto 3D structures.
Zero-Shot Segmentation and Tracking
By rendering a 3D object or scene from multiple viewpoints, researchers can generate dense 2D masks and map them back to the 3D surface.
- Rui Huang 0012 et al. introduced Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation Without Manual Labels, which leverages SAM to generate class-agnostic pseudo-labels via depth-based projection, allowing 3D models to inherit fine-grained segmentation capabilities.
- For dynamic environments, Aljosa Osep et al. developed Zero-Shot 4D Lidar Panoptic Segmentation, distilling knowledge from 2D video object segmentation models into the 4D LiDAR domain to track objects across time without manual point cloud tracking labels.
- To achieve real-time performance, Jiazhao Zhang et al. proposed OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging, which continuously merges 2D foundation model masks into a voxel-hashed 3D world model at 15 frames per second.
Open-Vocabulary Scene Graphs and Retrieval
By projecting text-aligned embeddings (like those from CLIP or OpenSeg) into 3D space, models can understand abstract natural language queries within physical environments.
- In PLA: Language-Driven Open-Vocabulary 3D Scene Understanding, Runyu Ding et al. used 2D images as a bridge to align 3D point cloud embeddings with rich textual descriptions, enabling open-vocabulary scene understanding.
- Qiao Gu et al. extended this to robotics in ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning, lifting 2D foundation model outputs into structured 3D graphs to let robots navigate using abstract commands like "find something healthy to eat."
- Similarly, Shiyang Lu et al. developed OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Data, fusing real-time 2D open-vocabulary detections into a persistent 3D memory bank for language-guided object retrieval.
Resolving the Noise and Viewpoint Inconsistency Problem
While cross-modal projection is elegant in theory, it faces a major hurdle in practice: 2D foundation models are inherently noisy and view-dependent. A 2D detector might label a chair correctly from a front view but misclassify or miss it entirely from a steep angle. When these inconsistent 2D predictions are back-projected into 3D space, they create conflicting labels for the exact same voxels or points.
Viewpoint A ───> [ 2D Model ] ───> Label: "Chair" ──┐
▼
[ 3D Voxel Fusion ] ───> Conflict!
▲
Viewpoint B ───> [ 2D Model ] ───> Label: "Table" ──┘
To make this distillation process work, researchers have developed sophisticated spatial-temporal consensus mechanisms:
Multi-View Voting and Voxel Filtering
Instead of trusting a single 2D projection, systems aggregate predictions from dozens of viewpoints. Pufan Zou et al. addressed this in AdaCo: Overcoming Visual Foundation Model Noise in 3D Semantic Segmentation via Adaptive Label Correction. Their framework refines initial pseudo-labels through inter-frame voxel voting and adaptively corrects labels during training by identifying the exact moment the model begins memorizing noise.
In LeAP: Consistent multi-domain 3D labeling using Foundation Models, Andras Palffy et al. combined a statistically grounded Bayesian voxel update rule with a 3D Consistency Network to smooth out the inherent noise of 2D-to-3D projections, allowing effective model training in niche domains like agricultural robotics.
Differentiable Rendering and Implicit Fields
Rather than performing hard geometric projections, some frameworks use differentiable rendering to enforce consistency. Adrian Hayler et al. demonstrated in S4C: Self-Supervised Semantic Scene Completion with Neural Fields that distilling 2D semantic features into an implicit neural field ensures temporal and spatial consistency across rendered views.
Similarly, Ang Cao et al. introduced LIFT-GS in From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs, leveraging 3D Gaussian Splatting and differentiable rendering to supervise 3D models with 2D foundation masks, effectively multiplying the utility of sparse 3D labels.
Training Native 3D Models Exclusively on 2D Data
The ultimate realization of this paradigm is the ability to train standard, high-performance 3D object detectors and segmentors without requiring any 3D sensor data or human labels during training.
By combining monocular depth estimation with 2D open-vocabulary models, pixels can be "lifted" into pseudo-3D training data. Rui Huang et al. proved this scalability in Training an Open-Vocabulary Monocular 3D Detection Model without 3D Data, showing that 3D object detectors trained solely on pseudo-labels from 2D images improve consistently as the volume of unlabeled image data scales up.
Taking this a step further, Timing Yang et al. developed ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images. Their framework establishes a bidirectional bridge—lifting 2D images into pseudo-3D space and rendering 3D point clouds back into textured 2D images—demonstrating that the semantic richness of 2D foundation models can entirely compensate for a total absence of native 3D annotations.
Key Foundational Work
Auto-Annotation of 3D Objects via ImageNet (2017) — contributed 2 concept elements to this direction.