In computer vision, there is a fundamental asymmetry between sensor modalities. Some sensors are inherently geometry-rich but operationally constrained: LiDAR provides highly accurate 3D point clouds but is expensive and bulky; stereo camera rigs resolve depth via geometric disparity but require precise calibration and double the data footprint. Conversely, other modalities are geometry-poor but highly practical: monocular RGB cameras, thermal sensors, and event cameras are cheap, lightweight, and ubiquitous, yet they suffer from fundamental spatial ambiguities (such as scale and depth ambiguity).
Cross-modal geometric distillation bridges this gap. It is a training paradigm that uses a privileged, geometry-aware teacher model to guide a constrained student model. By forcing the student to align its internal representations or outputs with the teacher's geometric priors, the student internalizes an implicit understanding of 3D structure. At inference time, the teacher sensor is completely discarded, leaving a high-performance, single-modality model capable of accurate spatial reasoning.
The Core Mechanism: Aligning Latent Spaces in Bird's-Eye View
To transfer knowledge from a 3D sensor (like LiDAR) to a 2D sensor (like a camera), the models must speak the same spatial language. Direct pixel-to-point matching is difficult because of the structural differences between dense image grids and sparse point clouds.
The solution is to project both modalities into a unified intermediate space—most commonly a Bird's-Eye-View (BEV) grid or a 3D voxel space. This shared canvas allows the geometric constraints of the teacher to directly supervise the semantic features of the student.
[Teacher: LiDAR Data] ───> [3D Voxel / BEV Space] <─── [Student: 2D Images]
│
(Geometry-Aware Distillation)
│
▼
[Student learns Spatial Priors]
As shown by Yanwei Li et al. in Unifying Voxel-based Representation with Transformer for 3D Object Detection, preserving the height dimension in a unified 3D voxel space prevents semantic ambiguity and allows a camera-only model to implicitly represent 3D structures. Similarly, Hao Shi et al. in CoBEV: Elevating Roadside 3D Object Detection with Depth and Height Complementarity demonstrate that supervising a monocular camera network with a multimodal teacher (LiDAR + camera) forces the student to approximate dense spatial representations.
This spatial alignment is so powerful that it can even recover information lost during network compression. Yandong Guo et al. in QD-BEV : Quantization-aware View-guided Distillation for Multi-view 3D Object Detection utilize camera extrinsics to create spatial masks that guide feature mapping. This geometry-aware distillation ensures that even heavily quantized, low-precision student models retain accurate coordinate associations and spatial awareness.
From Binocular Cues to Monocular Intuition
A classic application of this paradigm is transferring knowledge from stereo matching networks to monocular depth estimators. Stereo networks naturally resolve depth by finding correspondences between two offset views—a process strictly governed by projective geometry. Monocular networks, lacking a second viewpoint, must rely on contextual cues (like object size or perspective lines) to guess depth.
By treating a stereo network as a teacher, the monocular student can learn to infer depth without requiring explicit 3D ground-truth labels (like LiDAR scans). In Learning Monocular Depth by Distilling Cross-Domain Stereo Networks, Hongsheng Li et al. demonstrate that a stereo teacher provides dense, geometrically consistent labels that force a monocular student to focus on underlying structural depth rather than overfitting to domain-specific textures.
Qiulei Dong formalizes this relationship in Two-in-One Depth: Bridging the Gap Between Monocular and Binocular Self-supervised Depth Estimation, showing that distilling knowledge from a binocular path to a monocular path allows the single-image model to internalize geometric principles that are typically accessible only via explicit stereo correspondence.
Distilling Foundation Models for Specialized Sensors
The paradigm has evolved beyond physical sensors acting as teachers. Modern architectures leverage massive, pre-trained monocular depth foundation models (which have internalized robust geometric priors from millions of diverse images) to supervise specialized or lightweight networks.
This is particularly useful for unconventional sensors where labeled 3D data is scarce, such as event cameras. Event cameras track microsecond changes in brightness, making them excellent for high-speed motion but poor at capturing static scene geometry.
In Distil-E2D: Distilling Image-to-Depth Priors for Event-Based Monocular Depth Estimation, the authors mitigate the limitations of sparse sensor data by distilling dense, semantically rich supervision from image-to-depth foundation models into the event domain. Similarly, Pihai Sun in FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion uses latent space alignment to transfer geometric common sense from large-scale image models to event-based streams without requiring absolute ground-truth labels.
Handling Scale Ambiguity
When distilling from a foundation model, a key challenge is that these models often predict relative depth rather than absolute metric distance. To transfer structural intelligence without transferring scale errors, researchers alter the distillation objective.
As shown by Kyumin Hwang et al. in Scale-invariant and View-relational Representation Learning for Full Surround Monocular Depth, distilling the probability distribution of depth bins rather than raw depth values allows the student to capture the teacher's geometric expertise while remaining anchored to physical reality through separate, sparse metric supervision.
Why It Works: Internalizing Physical Structure
Cross-modal distillation works because it forces the student network to learn intermediate representations of physical reality as a prerequisite for making predictions. Instead of learning a direct, brittle mapping from 2D pixels to a 3D bounding box, the student is forced to mimic the teacher's understanding of spatial layout.
For example, the distillation process can force a network to learn whether a volume of space is empty or full. In Learning Occupancy for Monocular 3D Object Detection, Liang Peng et al. demonstrate that supervising a monocular vision model with sparse LiDAR-derived occupancy labels forces the network to learn underlying physical structures.
By learning these structural boundaries—such as the intersection of walls and ceilings, or the invariant shape profiles of CAD models (Georgakis et al. in Learning Local RGB-to-CAD Correspondences for Object Pose Estimation)—the student model stops relying solely on superficial color or texture cues. It begins leveraging the underlying geometry of the scene, achieving spatial reasoning capabilities that mimic the very sensors that were discarded after training.