In an ideal world, machine learning models would always have access to a rich tapestry of sensory inputs. To diagnose a patient, a model would read clinical records, analyze high-resolution MRI scans, and evaluate genetic markers simultaneously. To navigate a street, an autonomous vehicle would fuse real-time RGB video, LiDAR point clouds, and thermal imaging.
In reality, deployment environments are constrained. High-end sensors are expensive, power-hungry, or physically impractical for edge devices. Privacy regulations might forbid visible-light cameras in hospital bedrooms, forcing a reliance on thermal sensors. Weather conditions like heavy fog might blind optical cameras, leaving only radar or synthetic aperture radar (SAR).
This creates a fundamental asymmetry: we often have access to rich, multimodal data during training, but are restricted to a single, resource-constrained modality during inference.
Instead of throwing away auxiliary data or training a weak model from scratch, a powerful paradigm has emerged: leveraging multimodal relationships during training to bake "privileged" structural and semantic knowledge directly into a single-modality encoder. When deployed, the model operates entirely within its unimodal constraints, yet performs as if it still has its auxiliary senses intact.
The Core Intuition: Latent Space Geometry
How can a model learn to see what isn't there? The secret lies in using the auxiliary modality to reshape the latent space of the primary encoder during training.
Imagine trying to classify complex cardiac conditions using only standard, non-contrast "cine" Cardiac MRI sequences. The visual cues for tissue scarring (fibrosis) are incredibly subtle and easily missed. However, during training, we might also have access to Late Gadolinium Enhancement (LGE) images—a contrast-enhanced modality that makes fibrosis vividly clear.
Training Time:
[Cine MRI] ------> (Cine Encoder) -----\
>--- [Aligned Latent Space] ---> Contrastive Loss
[LGE Image] -----> (LGE Encoder) ------/
Inference Time:
[Cine MRI] ------> (Cine Encoder) ---------> [Informed Latent Space] ---> Diagnosis
If we train a cine-only model normally, it might fail to find the subtle patterns. But what if we use contrastive learning to align the latent representations of the cine images with their corresponding LGE images?
As demonstrated by Siqing Yuan et al. in LGE-Guided Cross-Modality Contrastive Learning for Gadolinium-Free Cardiomyopathy Screening in Cine CMR, this alignment forces the cine encoder to group its features in a way that mimics the clear, distinct clustering of the LGE space. The cine encoder learns to look for the exact, minute pixel variations that correlate with the contrast-enhanced structures. At inference time, the contrast agent and the LGE images are completely absent, but the cine encoder retains this geometric awareness, successfully identifying pathology on its own.
This strategy applies broadly across diverse data types. For instance, Paul Hager et al. developed a framework in Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data that bridges imaging and structured clinical records during pretraining. By forcing the image encoder to align with expert tabular data, the resulting unimodal image model achieves state-of-the-art diagnostic performance even when clinical records are unavailable at runtime.
Cross-Modal Knowledge Distillation
Another powerful mechanism for bridging the inference-time modality gap is cross-modal teacher-student distillation. Here, a "teacher" network is fed the privileged, high-fidelity modality, while a "student" network is fed the deployment modality. The student is penalized not just for making incorrect final predictions, but for failing to match the intermediate representations or attention maps of the teacher.
Consider 3D shape analysis. Processing 3D point clouds directly is computationally intensive and geometrically sparse. In Let Images Give You More: Point Cloud Cross-Modal Training for Shape Analysis, Shuguang Cui et al. utilize a 2D image teacher to guide a 3D point cloud student. Because 2D images possess dense texture and well-defined boundaries, the image teacher's attention maps act as a blueprint, teaching the 3D student where to focus its computational weight.
Similarly, in scenarios where hardware or environmental limits prevent the deployment of expensive sensors, adversarial distillation can bridge the gap. Nuno C. Garcia and Pietro Morerio introduced Adversarial Discriminative Modality Distillation (ADMD) in Learning with privileged information via adversarial discriminative modality distillation. Their framework allows an RGB-only model to "hallucinate" structural depth features at test time by training the RGB network to mimic a depth-sensor network through an adversarial loss.
Feature Hallucination and Memory Networks
Rather than just aligning latent spaces, some architectures explicitly "recall" or reconstruct the missing modality through dedicated memory modules or generative bridges.
For example, in audio-visual speech enhancement, capturing internal articulatory movements (like tongue position) can drastically improve performance in noisy environments. However, running an ultrasound machine during daily use is impossible. To solve this, Rui-Chen Zheng et al. proposed a lip-tongue memory network in Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement. During training, the network learns a paired dictionary of lip movements and ultrasound tongue images. At inference time, when given only standard video of a speaker's lips, the model queries its internal memory to recall and inject the missing tongue features into the speech enhancement pipeline.
Training: [Lip Video] + [Ultrasound Tongue Image] ---> Update Memory Matrix
|
Inference: [Lip Video] ----------------------------------> [Query Memory] ---> [Hallucinated Tongue Features] ---> Enhanced Speech
A similar approach is used in multispectral pedestrian detection. While visible (RGB) and thermal sensors provide complementary information across day and night, real-world systems might only have access to one stream at a time. In Towards Versatile Pedestrian Detector with Multisensory-Matching and Multispectral Recalling Memory, Jung Uk Kim et al. introduced a Multispectral Recalling Memory module. The system learns to store prototypical cross-modal features during training, allowing a single sensor stream at deployment to dynamically recall the missing sensor's characteristics.
Shifting the Paradigm
The traditional approach to multimodal learning treats missing data at inference as a failure state or a missing-value problem to be solved via imputation. The papers highlighted here represent a paradigm shift: multimodality is treated as a training-time catalyst rather than an inference-time requirement.
By framing auxiliary sensors as "privileged information," these techniques allow us to build models that are lightweight, privacy-preserving, and cheap to deploy, without sacrificing the deep, cross-modal insights that can only be uncovered in a fully equipped laboratory or data-rich training environment.
Key Foundational Work
Auxiliary Training Information Assisted Visual Recognition (2015) — contributed 1 concept elements to this direction.
Learning with Side Information through Modality Hallucination (2016) — contributed 1 concept elements to this direction.
Learning from Multiple Sources for Video Summarisation (2015) — contributed 1 concept elements to this direction.