Survival Prediction of Brain Cancer with Incomplete Radiology, Pathology, Genomic, and Demographic Data

Can Cui, Han Liu, Quan Liu, Ruining Deng, Zuhayr Asad, Yaohong Wang, Shilin Zhao, Haichun Yang, Bennett A. Landman, Yuankai Huo

TL;DR

This paper tackles the challenge of predicting brain cancer survival using multiple data types (radiology, pathology, genomics, demographics) when some of this data is missing for a patient. They introduce a novel "Multi-modal Learning with Missing Data" (MMD) pipeline that effectively leverages available information, achieving improved survival prediction accuracy compared to existing methods.

What Problem Does This Paper Address?

Accurately predicting brain cancer survival is crucial for effective treatment planning. Traditionally, this involves integrating information from various sources like radiology scans, pathology slides, genetic data, and patient demographics. However, in real-world clinical settings, it's common for one or more of these data modalities to be incomplete for a given patient. Existing multi-modal learning approaches often assume complete data, leading to two main issues: (1) discarding patients with missing data exacerbates data scarcity and increases the risk of overfitting, and (2) models trained on complete data may not generalize well to real-world scenarios with missing information. This paper aims to bridge this gap by developing a method that can robustly predict survival even when faced with incomplete multi-modal data.

Methods

The core of this paper is the proposed Multi-modal Learning with Missing Data (MMD) pipeline. It's a two-stage approach:

  1. Uni-modal Feature Embedding: This stage involves training independent deep neural networks for each data modality (radiology, pathology, genomics, demographics) to extract meaningful low-dimensional features. For image modalities (radiology and pathology), they leverage pre-trained networks like ResNet-18 and Vgg-16, with specific strategies for handling input dimensionality and image patching. For genomic and demographic data, they use self-normalized networks to mitigate overfitting.
  2. Multi-modal Information Fusion: This is where the novelty lies, particularly in handling missing data.
  3. Mean Vector Fusion: They adopt a "mean vector" (element-wise average) approach for fusing features from available modalities. This is chosen for its "order-less" nature, making it more flexible with missing data compared to concatenation or tensor fusion.
  4. Modality Dropout: During training, they randomly drop out modalities to encourage the model to learn robust representations from incomplete sets of data.
  5. Modality Reconstruction: To further improve fusion, they incorporate a reconstruction loss. The model is trained to reconstruct not only the fused representation but also the missing uni-modal embeddings, forcing it to learn cross-modality relationships.

The overall loss function combines a Cox loss for survival prediction with the reconstruction loss.

Datasets

The study utilizes a large-scale dataset by combining data from three sources:

  • The Cancer Genome Atlas (TCGA) [1, 2]
  • The Cancer Imaging Archive (TCIA) [13]
  • The BraTs dataset [5, 6, 20]

This combined dataset includes data from 962 patients, with each patient having at least one modality available. The paper notes that this dataset is made publicly available.

Key Insights and Ideas

  • Missing Data is the Norm, Not the Exception: The paper strongly emphasizes that real-world clinical data is rarely complete, and models must be designed to handle this reality rather than assuming perfect data.
  • Leveraging Incomplete Data is Better Than Discarding It: By developing methods to utilize available modalities even when others are missing, the MMD pipeline significantly improves prediction accuracy compared to methods that would simply exclude patients with incomplete data.
  • Mean Vector Fusion's Flexibility: The choice of mean vector fusion is highlighted for its inherent robustness to missing modalities, making it a suitable backbone for this problem.
  • Modality Dropout and Reconstruction as Synergistic Techniques: Combining modality dropout during training with a reconstruction loss for missing modalities proves to be an effective strategy for learning more generalized and accurate representations.
  • Two-Stage Training is Competitive and Efficient: The paper finds that a two-stage training approach (pre-training uni-modal encoders and then training the fusion model) is competitive with end-to-end training and offers better memory efficiency.

Key Results

The proposed MMD method achieved a significant improvement in survival prediction, increasing the C-index from 0.7624 to 0.8053 when compared to prior benchmarks. Specifically, Table 1 shows MMD outperforming Pathomic Fusion [12] and Deep Orthogonal Fusion [10]. Table 3 further details the performance gains across various training strategies and fusion methods, demonstrating that the MMD approach, particularly with modality dropout and reconstruction, consistently improves performance on testing sets with missing data. The study also found that finetuning only the last layers of pre-trained networks for uni-modal feature extraction yielded better results than finetuning the entire network.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their method improves performance with missing data, the C-index is still lower on testing sets with missing modalities compared to testing sets with complete modalities. The specific rate of modality dropout (0.5) was empirically selected.
  • Future Work: The authors suggest applying their method to external datasets to further validate its generalizability. They also propose exploring other uni-modal embedding methods to potentially enhance feature extraction.

Summary updated

Abstract

Integrating cross-department multi-modal data (e.g., radiology, pathology, genomic, and demographic data) is ubiquitous in brain cancer diagnosis and survival prediction. To date, such an integration is typically conducted by human physicians (and panels of experts), which can be subjective and semi-quantitative. Recent advances in multi-modal deep learning, however, have opened a door to leverage such a process in a more objective and quantitative manner. Unfortunately, the prior arts of using four modalities on brain cancer survival prediction are limited by a “complete modalities” setting (i.e., with all modalities available). Thus, there are still open questions on how to effectively predict brain cancer survival from incomplete radiology, pathology, genomic, and demographic data (e.g., one or more modalities might not be collected for a patient). For instance, should we use both complete and incomplete data, and more importantly, how do we use such data? To answer the preceding questions, we generalize the multi-modal learning on cross-department multi-modal data to a missing data setting. Our contribution is three-fold: 1) We introduce a multi-modal learning with missing data (MMD) pipeline with competitive performance and less hardware consumption; 2) We extend multi-modal learning on radiology, pathology, genomic, and demographic data into missing data scenarios; 3) A large-scale public dataset (with 962 patients) is collected to systematically evaluate glioma tumor survival prediction using four modalities. The proposed method improved the C-index of survival prediction from 0.7624 to 0.8053.
BibTeX
@inproceedings{cui2022survival,
  author = {Can Cui and Han Liu and Quan Liu and Ruining Deng and Zuhayr Asad and Yaohong Wang and Shilin Zhao and Haichun Yang and Bennett A. Landman and Yuankai Huo},
  title = {{Survival Prediction of Brain Cancer with Incomplete Radiology, Pathology, Genomic, and Demographic Data}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  doi = {10.1007/978-3-031-16443-9\_60},
}
Figure