TL;DR
This paper tackles the challenge of predicting brain cancer survival using multiple data types (radiology, pathology, genomics, demographics) when some of this data is missing for a patient. They introduce a novel "Multi-modal Learning with Missing Data" (MMD) pipeline that effectively leverages available information, achieving improved survival prediction accuracy compared to existing methods.
What Problem Does This Paper Address?
Accurately predicting brain cancer survival is crucial for effective treatment planning. Traditionally, this involves integrating information from various sources like radiology scans, pathology slides, genetic data, and patient demographics. However, in real-world clinical settings, it's common for one or more of these data modalities to be incomplete for a given patient. Existing multi-modal learning approaches often assume complete data, leading to two main issues: (1) discarding patients with missing data exacerbates data scarcity and increases the risk of overfitting, and (2) models trained on complete data may not generalize well to real-world scenarios with missing information. This paper aims to bridge this gap by developing a method that can robustly predict survival even when faced with incomplete multi-modal data.
Methods
The core of this paper is the proposed Multi-modal Learning with Missing Data (MMD) pipeline. It's a two-stage approach:
- Uni-modal Feature Embedding: This stage involves training independent deep neural networks for each data modality (radiology, pathology, genomics, demographics) to extract meaningful low-dimensional features. For image modalities (radiology and pathology), they leverage pre-trained networks like ResNet-18 and Vgg-16, with specific strategies for handling input dimensionality and image patching. For genomic and demographic data, they use self-normalized networks to mitigate overfitting.
- Multi-modal Information Fusion: This is where the novelty lies, particularly in handling missing data.
- Mean Vector Fusion: They adopt a "mean vector" (element-wise average) approach for fusing features from available modalities. This is chosen for its "order-less" nature, making it more flexible with missing data compared to concatenation or tensor fusion.
- Modality Dropout: During training, they randomly drop out modalities to encourage the model to learn robust representations from incomplete sets of data.
- Modality Reconstruction: To further improve fusion, they incorporate a reconstruction loss. The model is trained to reconstruct not only the fused representation but also the missing uni-modal embeddings, forcing it to learn cross-modality relationships.
The overall loss function combines a Cox loss for survival prediction with the reconstruction loss.
Datasets
The study utilizes a large-scale dataset by combining data from three sources:
- The Cancer Genome Atlas (TCGA) [1, 2]
- The Cancer Imaging Archive (TCIA) [13]
- The BraTs dataset [5, 6, 20]
This combined dataset includes data from 962 patients, with each patient having at least one modality available. The paper notes that this dataset is made publicly available.
Key Insights and Ideas
- Missing Data is the Norm, Not the Exception: The paper strongly emphasizes that real-world clinical data is rarely complete, and models must be designed to handle this reality rather than assuming perfect data.
- Leveraging Incomplete Data is Better Than Discarding It: By developing methods to utilize available modalities even when others are missing, the MMD pipeline significantly improves prediction accuracy compared to methods that would simply exclude patients with incomplete data.
- Mean Vector Fusion's Flexibility: The choice of mean vector fusion is highlighted for its inherent robustness to missing modalities, making it a suitable backbone for this problem.
- Modality Dropout and Reconstruction as Synergistic Techniques: Combining modality dropout during training with a reconstruction loss for missing modalities proves to be an effective strategy for learning more generalized and accurate representations.
- Two-Stage Training is Competitive and Efficient: The paper finds that a two-stage training approach (pre-training uni-modal encoders and then training the fusion model) is competitive with end-to-end training and offers better memory efficiency.
Key Results
The proposed MMD method achieved a significant improvement in survival prediction, increasing the C-index from 0.7624 to 0.8053 when compared to prior benchmarks. Specifically, Table 1 shows MMD outperforming Pathomic Fusion [12] and Deep Orthogonal Fusion [10]. Table 3 further details the performance gains across various training strategies and fusion methods, demonstrating that the MMD approach, particularly with modality dropout and reconstruction, consistently improves performance on testing sets with missing data. The study also found that finetuning only the last layers of pre-trained networks for uni-modal feature extraction yielded better results than finetuning the entire network.
Limitations and Future Work
- Limitations: The paper acknowledges that while their method improves performance with missing data, the C-index is still lower on testing sets with missing modalities compared to testing sets with complete modalities. The specific rate of modality dropout (0.5) was empirically selected.
- Future Work: The authors suggest applying their method to external datasets to further validate its generalizability. They also propose exploring other uni-modal embedding methods to potentially enhance feature extraction.
Summary updated