The field of 3D computer vision faces a chronic data scarcity problem. While we have billions of labeled 2D images and massive vision-language models like CLIP, 3D point clouds—the sparse, irregular sets of coordinates captured by LiDAR—are expensive to collect and label.
To solve this, researchers are developing methods to "bridge the modality gap," allowing 3D models to inherit the rich semantic intelligence of 2D foundation models. This is not a simple matter of copying weights; it requires overcoming a fundamental structural discrepancy between the dense, regular grid of pixels and the sparse, unordered nature of spatial points.
The Structural Discrepancy
The primary barrier to cross-modal learning is that 2D and 3D data speak different mathematical languages. As Jiaxin Chen and Yi Fang note in Deep Cross-Modality Adaptation via Semantics Preserving Adversarial Learning for Sketch-Based 3D Shape Retrieval, 2D representations are often stroke-based or grid-based, while 3D geometries are volumetric or point-based.
This "modality gap" means that a feature extractor trained to recognize a "chair" in a photo cannot naturally recognize the same chair in a LiDAR scan. The goal of this research direction is to create a shared latent space where the semantic concept of "chair" is invariant to whether the input is a pixel or a point.
Strategy 1: Projection and Rendering
The most intuitive way to bridge this gap is to turn the 3D problem into a 2D one. By projecting 3D point clouds into multi-view 2D images or depth maps, we can use "off-the-shelf" 2D models directly.
In PointCLIP: Point Cloud Understanding by CLIP, Renrui Zhang and colleagues demonstrated that projecting points into multi-view depth maps allows the CLIP model to perform zero-shot 3D classification without any 3D-specific training. Similarly, Zuo Zuo and Jiahao Dong used this multi-view generation in CLIP3D-AD to detect industrial anomalies, effectively transferring CLIP’s massive 2D knowledge to 3D defect detection.
Strategy 2: Cross-Modal Distillation
Rather than converting the data, we can convert the knowledge. In a teacher-student framework, a powerful 2D model (the teacher) guides a 3D model (the student) to produce similar feature representations for the same object.
The ACT (Autoencoders as Cross-Modal Teachers) framework by Runpei Dong and Linfeng Zhang (Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?) distills "dark knowledge" from image Transformers into 3D students using masked point modeling. This allows the 3D model to learn high-level semantics that are usually only present in massive 2D datasets. More recently, CleverDistiller simplified this process by using a projection head and an occupancy prediction task to align 3D LiDAR backbones with 2D models like DINOv2, achieving significant gains in semantic segmentation.
Strategy 3: Structural Alignment via Tokenization
A more direct approach is to force the 3D data to "look" like 2D data at the input level. This involves creating 3D tokenizers that produce embeddings compatible with the input space of frozen 2D Transformers.
Xiaoshui Huang and Tong He proposed EPCL (Efficient Point Cloud Learning), which uses a lightweight tokenizer to plug a frozen 2D CLIP transformer into a 3D pipeline. By aligning the 3D tokenizer with the 2D transformer's expected manifold, they achieved state-of-the-art results while training only a fraction of the parameters. Similarly, Pix4Point by Guocheng Qian uses a Relative Progressive Tokenizer to allow standard ImageNet-pretrained Transformers to process 3D coordinates.
Applications in Registration and Retrieval
This cross-modal alignment is critical for tasks where 2D and 3D data must interact in real-time, such as Image-to-Point Cloud (I2P) registration.
In CorrI2P, Siyu Ren and Junhui Hou tackle the difficulty of matching color-based pixels with geometric spatial points. By establishing dense correspondences, they allow autonomous vehicles to localize a 2D camera image within a 3D LiDAR map. GraphI2P further refines this by converting images into "virtual point clouds" and using graph learning to prune false matches, bridging the gap between texture and geometry.
Why It Works: Semantic Invariance
The underlying intuition across all these papers—from Point2Real to MVNet—is that semantics are modality-agnostic. A "stop sign" has the same functional meaning whether it is represented by a cluster of red pixels or a flat vertical plane of LiDAR points. By using 2D models as a bridge, researchers are essentially teaching 3D models to ignore the "noise" of the specific sensor and focus on the underlying geometric and semantic structure of the world.