Introduction
In clinical oncology, diagnostic data is rarely complete. While an ideal dataset might include genomic sequences, high-resolution CT and MRI scans, and pathology slides for every patient, the reality is far messier. Medical records are frequently riddled with "holes"—a patient may have a CT scan but lack an MRI, or possess genomic data without corresponding digital pathology.
Most existing machine learning models struggle with this "missing modality" problem, often discarding incomplete records or defaulting to single-modality analysis. To address this, the authors introduce MMIST-ccRCC, a curated, multi-modal dataset of 618 patients with clear cell renal cell carcinoma, alongside a robust framework designed to "hallucinate" missing data in latent space.
The MMIST-ccRCC Dataset
Curated from three major public repositories (TCGA, TCIA, and CPTAC), MMIST-ccRCC provides a standardized benchmark for multi-modal systems. The dataset includes five primary modalities:
- Clinical Data: 11 variables.
- Genomics: 3 key mutations (VHL, PBRM1, TTN).
- Histopathology: 2,573 Whole Slide Images (WSIs).
- Radiology: 736 CT volumes and 552 MRI volumes.
Crucially, the dataset preserves real-world missingness. While all patients have clinical data and WSIs, only 74% have genomics, 40% have CTs, and a mere 8% have MRIs. This sparsity reflects the actual challenges faced in clinical data integration.
Methodology
The researchers developed a multi-stage pipeline to predict 12-month survival, centered on handling incomplete records without discarding data.
- Feature Extraction: Domain-specific models were used for each type. WSIs were processed via CLAM (ResNet50-based), while 3D radiology scans were encoded using MeD-3D (3D-ResNet18).
- MIL Selection: Since patients often have multiple scans, a Multiple Instance Learning (MIL) framework automatically selects the most representative image based on its predictive utility.
- Latent Reconstruction: This is the core innovation. Using an encoder-decoder architecture, the model infers the latent features of missing modalities based on available ones. For example, if an MRI is missing, the model uses CT and clinical data to estimate the missing MRI features.
- Fusion Strategies: The team compared Early Fusion (combining features before classification) and Late Fusion (averaging individual modality predictions).
Key Results
Performance was measured using Balanced Accuracy (BAcc) to account for the high survival rate (88%) in the cohort. The results clearly demonstrate the value of both multi-modal integration and latent reconstruction.
- Single Modality Baseline: The strongest single-modality predictor (Clinical + Genomics) reached 73.16% BAcc.
- Standard Multi-Modal Fusion: Early fusion improved this to 82.32% BAcc.
- Fusion + Reconstruction: The proposed method (Early Fusion with Latent Reconstruction) achieved the top score of 84.70% BAcc.
Notably, for the subset of patients who did have CT scans, the reconstruction-augmented model boosted accuracy from approximately 61% (MIL alone) to over 90%.
Discussion and Future Work
The study reveals that even highly scarce data (like the 8% MRI subset) can provide complementary information that improves overall model performance when properly integrated. The "hallucination" of latent features proved more effective than standard fusion techniques, as it allows the model to leverage cross-modal relationships that are otherwise lost in incomplete records.
Future work will focus on expanding the genomic depth beyond three genes and exploring more advanced spatial encoding for imaging data. The MMIST-ccRCC dataset and associated code are publicly available to encourage further research into robust multi-modal medical systems.
Summary updated