MMIST-ccRCC: A Real World Medical Dataset for the Development of Multi-Modal Systems

Tiago Mota, Maria Rita Verdelho, Diogo J. Araújo, Alceu Bissoto, Carlos Santiago, Catarina BarataWorkshops @ CVPR 2024 · DOI · arXiv

Introduction

In clinical oncology, diagnostic data is rarely complete. While an ideal dataset might include genomic sequences, high-resolution CT and MRI scans, and pathology slides for every patient, the reality is far messier. Medical records are frequently riddled with "holes"—a patient may have a CT scan but lack an MRI, or possess genomic data without corresponding digital pathology.

Most existing machine learning models struggle with this "missing modality" problem, often discarding incomplete records or defaulting to single-modality analysis. To address this, the authors introduce MMIST-ccRCC, a curated, multi-modal dataset of 618 patients with clear cell renal cell carcinoma, alongside a robust framework designed to "hallucinate" missing data in latent space.

The MMIST-ccRCC Dataset

Curated from three major public repositories (TCGA, TCIA, and CPTAC), MMIST-ccRCC provides a standardized benchmark for multi-modal systems. The dataset includes five primary modalities:

  • Clinical Data: 11 variables.
  • Genomics: 3 key mutations (VHL, PBRM1, TTN).
  • Histopathology: 2,573 Whole Slide Images (WSIs).
  • Radiology: 736 CT volumes and 552 MRI volumes.

Crucially, the dataset preserves real-world missingness. While all patients have clinical data and WSIs, only 74% have genomics, 40% have CTs, and a mere 8% have MRIs. This sparsity reflects the actual challenges faced in clinical data integration.

Methodology

The researchers developed a multi-stage pipeline to predict 12-month survival, centered on handling incomplete records without discarding data.

Overview of the MMIST-ccRCC pipeline, featuring MIL selection and latent reconstruction.
  1. Feature Extraction: Domain-specific models were used for each type. WSIs were processed via CLAM (ResNet50-based), while 3D radiology scans were encoded using MeD-3D (3D-ResNet18).
  2. MIL Selection: Since patients often have multiple scans, a Multiple Instance Learning (MIL) framework automatically selects the most representative image based on its predictive utility.
  3. Latent Reconstruction: This is the core innovation. Using an encoder-decoder architecture, the model infers the latent features of missing modalities based on available ones. For example, if an MRI is missing, the model uses CT and clinical data to estimate the missing MRI features.
  4. Fusion Strategies: The team compared Early Fusion (combining features before classification) and Late Fusion (averaging individual modality predictions).

Key Results

Performance was measured using Balanced Accuracy (BAcc) to account for the high survival rate (88%) in the cohort. The results clearly demonstrate the value of both multi-modal integration and latent reconstruction.

Comparison of survival prediction performance (Balanced Accuracy) across different fusion strategies.
  • Single Modality Baseline: The strongest single-modality predictor (Clinical + Genomics) reached 73.16% BAcc.
  • Standard Multi-Modal Fusion: Early fusion improved this to 82.32% BAcc.
  • Fusion + Reconstruction: The proposed method (Early Fusion with Latent Reconstruction) achieved the top score of 84.70% BAcc.

Notably, for the subset of patients who did have CT scans, the reconstruction-augmented model boosted accuracy from approximately 61% (MIL alone) to over 90%.

Discussion and Future Work

The study reveals that even highly scarce data (like the 8% MRI subset) can provide complementary information that improves overall model performance when properly integrated. The "hallucination" of latent features proved more effective than standard fusion techniques, as it allows the model to leverage cross-modal relationships that are otherwise lost in incomplete records.

Future work will focus on expanding the genomic depth beyond three genes and exploring more advanced spatial encoding for imaging data. The MMIST-ccRCC dataset and associated code are publicly available to encourage further research into robust multi-modal medical systems.

Summary updated

BibTeX
@inproceedings{mota2024mmist,
  author = {Tiago Mota and Maria Rita Verdelho and Diogo J. Araújo and Alceu Bissoto and Carlos Santiago and Catarina Barata},
  title = {{MMIST-ccRCC: A Real World Medical Dataset for the Development of Multi-Modal Systems}},
  booktitle = {Workshops @ CVPR},
  year = {2024},
  doi = {10.1109/cvprw63382.2024.00246},
}
Figure