Multi-modal Data Binding for Survival Analysis Modeling with Incomplete Data and Annotations

Linhao Qu, Dan Huang, Shaoting Zhang, Xiaosong Wang

Abstract

Survival analysis in oncology is frequently hindered by missing data modalities and right-censored patient records. This paper presents a novel multi-modal data binding framework that leverages specialized medical foundation models and a progressive survival disambiguation technique. By aligning diverse data types—including pathology, radiology, and clinical text—into a unified representation space, the proposed method achieves state-of-the-art performance in cancer survival prediction even when data is incomplete or annotations are censored.

Introduction

Predicting patient survival or disease-free intervals is critical for personalized cancer treatment. While multi-modal data integration (pathology slides, radiology images, clinical notes) offers superior predictive power, real-world clinical datasets are often incomplete. Patients may lack specific scans or reports, and many remain alive at the study's conclusion (right-censoring). Traditional models often fail to handle missing modalities effectively or treat censored data as simple binary outcomes, leading to biased risk assessments. This research addresses these gaps by "binding" disparate data types and modeling the latent risk of censored patients.

Methodology

The proposed framework employs a multi-step approach to integrate heterogeneous clinical data and handle missing information.

Modality-Specific Foundation Models

Rather than using general-purpose AI, the framework utilizes domain-specific foundation models: PLIP for pathology, MedSAM-3D for radiology, and BioLinkBERT for clinical text. These models are better equipped to capture the specific nuances of medical data compared to general vision-language models like CLIP.

Attention-based Aggregation and Contrastive Alignment

An attention mechanism is used to weigh the importance of various data instances (e.g., specific tumor patches in a whole-slide image) before fusion. To handle missing modalities, the model employs Patient-wise Contrastive Alignment. By aligning features from different modalities in a shared embedding space, the model learns a robust representation that remains effective even if certain data types are absent during testing.

Progressive Survival Disambiguation Learning

To address right-censoring, the model generates "pseudo labels" for censored patients by estimating their future risk. A "warm-up" function allows the model to gradually incorporate these predictions as its confidence grows during training, capturing a more realistic hazard profile than standard survival models.

Overview of the Multi-modal Data Binding framework architecture.

Experimental Results

The framework was evaluated on two in-house colorectal cancer datasets (367 and 193 patients, respectively) characterized by real-world "incompleteness," where radiology data and specific clinical notes were intermittently absent.

Performance Comparison

The model outperformed existing state-of-the-art (SOTA) methods, such as MMD and ShaSpec, across Overall Survival (OS) and Disease-Free Survival (DFS) tasks. On Dataset 2, it achieved a Concordance Index (CI) of 0.914 for OS, surpassing the previous SOTA of 0.905. It also consistently demonstrated lower Brier Scores (BS), indicating superior probability calibration.

Clinical Utility and Ablation

Kaplan-Meier analysis confirmed the model's ability to stratify patients into high-risk and low-risk groups with high statistical significance ($p < 1e-7$). Ablation studies highlighted that specialized medical foundation models (e.g., PLIP) significantly outperform general-purpose alternatives, and the contrastive alignment strategy is vital for maintaining performance with missing data.

Performance metrics and Kaplan-Meier survival analysis.

Conclusion

This work demonstrates that specialized medical foundation models and innovative label disambiguation techniques can effectively overcome the challenges of missing data and censoring in survival analysis. By creating a robust, aligned representation of multi-modal clinical data, the framework provides more accurate and clinically relevant risk predictions for cancer patients. Future work will focus on validating these findings across larger, multi-center public datasets to ensure generalizability.

Summary updated

BibTeX
@inproceedings{qu2024multi,
  author = {Linhao Qu and Dan Huang and Shaoting Zhang and Xiaosong Wang},
  title = {{Multi-modal Data Binding for Survival Analysis Modeling with Incomplete Data and Annotations}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  doi = {10.1007/978-3-031-72086-4\_47},
}
Figure