Multi-modal machine learning operates on a powerful premise: combining complementary data streams yields a more holistic representation of an underlying phenomenon. In clinical diagnostics, for instance, a complete suite of MRI sequences (such as T1, T2, FLAIR, and contrast-enhanced T1c) allows a model to delineate tumor boundaries with high precision.
However, this ideal scenario routinely collapses in practice. Financial costs, scan corruption, patient discomfort, and varying hospital protocols mean that real-world datasets are fundamentally incomplete. Standard deep learning architectures are brittle under these conditions; they expect a fixed, deterministic set of inputs and fail catastrophically when an expected modality is missing.
To bridge this gap, researchers have developed techniques that allow models to remain highly accurate even when critical data channels vanish. The core insight driving this field is cross-modal redundancy: different modalities are not independent random variables, but distinct physical or biological reflections of the same underlying ground truth. This shared information structure allows models to either synthesize the missing data or align variable inputs into a unified space.
Paradigm 1: Generative Cross-Modality Synthesis
One intuitive way to handle a missing modality is to simply "fill in the blank" before passing the data to a downstream model. Because different modalities share underlying anatomical geometry, generative models can exploit these correlations to translate available scans into missing ones.
Early frameworks like Modality Propagation, introduced in Modality Propagation: Coherent Synthesis of Subject-Specific Scans with Data-Driven Regularization by Dong Hye Ye et al., generalized patch-based label propagation to continuous image values to synthesize patient-specific scans. With the advent of deep generative networks, this translation became significantly more sophisticated. In Auto-GAN: Self-Supervised Collaborative Learning for Medical Image Synthesis, Bing Cao 0002 et al. utilized a self-supervised teacher network to provide feature-level guidance, allowing a generator to leverage all available scans simultaneously to reconstruct missing sequences without structural deformation.
To capture long-range structural dependencies and avoid the spatial blurring common in standard Generative Adversarial Networks (GANs), Onat Dalmaz et al. proposed ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis. ResViT combines the local precision of convolutional layers with the global context-awareness of Vision Transformers.
More recently, diffusion models have pushed the boundaries of synthesis fidelity. In Unsupervised Medical Image Translation with Adversarial Diffusion Models, Muzaffer Özbey et al. developed SynDiff, which pairs a source-conditional adversarial projector with a diffusion process. This allows the model to synthesize high-quality target modalities (such as a T2 MRI from a T1 scan) in a fraction of the steps required by standard diffusion models, making real-time clinical imputation feasible.
[Available Modalities] ───► [Generative Model (GAN/Diffusion)] ───► [Synthesized Modality]
│
▼
[Downstream Task] ◄─────────────────────────────────────────────────────────┘
Paradigm 2: Shared Latent Spaces and Permutation-Invariant Fusion
While generative synthesis is visually compelling, it introduces a dangerous point of failure: the model might hallucinate pathological features or introduce artifacts that mislead clinical decisions. An alternative paradigm bypasses explicit image generation entirely, opting instead to project all available modalities into a unified, shared latent space.
The seminal blueprint for this approach was established by Mohammad Havaei et al. in HeMIS: Hetero-Modal Image Segmentation. Instead of concatenating raw inputs, HeMIS encodes each available modality independently into a latent representation. Crucially, these representations are fused using simple, permutation-invariant statistics—specifically, the mean and variance across the available latent vectors.
$$\mu = \frac{1}{|M_{avail}|} \sum_{m \in M_{avail}} E_m(X_m)$$
Because the mathematical formulation of a mean or variance calculation does not depend on the number of inputs, the network's downstream layers receive a tensor of identical shape regardless of which modalities are missing. As a result, the model's performance degrades gracefully rather than crashing when data is removed.
Building on this foundation, recent architectures replace simple statistical pooling with attention mechanisms to dynamically weight the available inputs based on their informational value. For example, Xin Yu et al. developed RFNet: Region-aware Fusion Network for Incomplete Multi-modal Brain Tumor Segmentation, which utilizes a region-aware fusion module to weight available scans based on their specific strengths for different anatomical zones.
To better capture global interactions across modalities, Yao Zhang 0010 et al. introduced mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation, leveraging a Transformer architecture to bridge local convolutional details with global multi-modal context.
Modality A ──► [Encoder A] ──┐
├──► [Permutation-Invariant Pooling] ──► [Shared Latent Space] ──► [Task Output]
Modality B ──► [Encoder B] ──┘ (Mean / Variance / Attention)
[Missing] ──► [ - ]
To optimize the heavy computational footprint of these multi-encoder systems, Changjian Wang proposed IMS²Trans in Scalable Swin Transformer network for brain tumor segmentation from incomplete MRI modalities. The author demonstrated that a single, shared-weight Swin Transformer encoder could replace multiple modality-specific encoders, drastically reducing parameter overhead while maintaining robust performance across arbitrary missing-data patterns.
Paradigm 3: Cross-Modal Knowledge Distillation and Co-Training
A third strategy focuses on the training dynamics themselves, forcing available modalities to act as "surrogates" or "teachers" for the missing ones. The goal is to ensure that the network learns representations that are robust to missing data by exploiting cross-modal dependencies during optimization.
In ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalities, Yixin Wang et al. proposed a dual-path co-training strategy. By applying information theory constraints, the network is forced to transfer knowledge from a full-modality path to a missing-modality path, effectively teaching the available data channels to "recover" the physiological signatures of the absent ones.
Similarly, Delin Chen et al. introduced a group self-support learning framework (GSS) in Scratch Each Other's Back: Incomplete Multi-modal Brain Tumor Segmentation Via Category Aware Group Self-Support Learning. During training, different modalities dynamically vote and teach one another, identifying which available scan acts as the "expert" for a given region and transferring that expertise to the rest of the network.
This concept is particularly crucial when dealing with highly imbalanced datasets. As Junjie Shi et al. noted in PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Rates, real-world clinical data often exhibits highly skewed missing rates, where certain expensive scans are exceptionally rare. By employing preference-aware self-distillation, their framework prevents dominant, ubiquitous modalities from overshadowing rarer ones, ensuring the network remains sensitive to subtle features even when only a subset of inputs is available at deployment.
Beyond Centralized Imaging: Federated and Multi-Omic Frontiers
The challenge of missing modalities extends far beyond centralized medical imaging pipelines. In modern healthcare networks, data is often siloed across different institutions, each utilizing distinct imaging protocols. To address this, Dong Wei et al. developed FedMEMA in Federated Modality-Specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation. This federated learning framework uses modality-specific encoders and centralized "multimodal anchors" to calibrate local models across sites, effectively allowing hospitals to benefit from representations of modalities they do not possess without sharing raw patient data.
Furthermore, the problem scales up significantly when moving from imaging to multi-omics, where missing data channels can span genomic, transcriptomic, and clinical boundaries. Frameworks like CustOmics, detailed in CustOmics: A versatile deep-learning based strategy for multi-omics integration by Hakim Benkirane et al., adapt these latent alignment strategies to high-dimensional biological data, ensuring that precision oncology models remain viable even when expensive genomic profiles are missing.
By shifting the focus from rigid, multi-channel inputs to flexible latent spaces, generative translation, and cross-modal distillation, this research direction transforms multi-modal deep learning from an idealized laboratory tool into a resilient framework capable of navigating the messy, incomplete realities of clinical data.
Key Foundational Work
Modality Propagation: Coherent Synthesis of Subject-Specific Scans with Data-Driven Regularization (2013) — contributed 1 concept elements to this direction.