Abstract
The Cancer Genome Atlas (TCGA) has enabled novel discoveries and served as a large-scale reference through its harmonized genomics, clinical, and image data. Prior studies have trained bespoke cancer survival prediction models from unimodal or multimodal TCGA data. A modern paradigm in biomedical deep learning is the development of foundation models (FMs) to derive meaningful feature embeddings, agnostic to a specific modeling task. Biomedical text especially has seen growing development of FMs. While TCGA contains free-text data as pathology reports, these have been historically underutilized. Here, we investigate the feasibility of training classical, multimodal survival models over zero-shot embeddings extracted by FMs. We show the ease and additive effect of multimodal fusion, outperforming unimodal models. We demonstrate the benefit of including pathology report text and rigorously evaluate the effect of model-based text summarization and hallucination. Overall, we modernize survival modeling by leveraging FMs and information extraction from pathology reports.
Introduction
Predicting cancer survival is a cornerstone of precision medicine, yet it requires the integration of heterogeneous data types: tabular clinical records, high-dimensional gene expression (RNA-seq), massive pathology slides, and unstructured free-text reports. Historically, researchers have relied on "bespoke" deep learning models—complex, custom architectures trained from scratch for specific tasks. These models are often difficult to scale, hard to combine, and frequently omit pathology reports due to the challenges of processing scanned PDFs and noisy OCR text.
This research demonstrates a shift from custom architecture engineering toward an embedding-centric paradigm. By leveraging zero-shot foundation models (FMs), the authors show that simple statistical models can achieve state-of-the-art multimodal fusion, significantly improving survival predictions across 32 cancer types.
Methodology: A Modular Embedding Framework
The researchers developed a modular framework that treats different data modalities as "pluggable" embeddings. This approach bypasses the need for task-specific training by using pre-trained FMs to generate vector representations for each data type:
- Images: UNI2-h for histopathology slides.
- Genomics: BulkRNABert and UCE for RNA-seq data.
- Text: BioMistral for pathology reports, following a summarization step using Llama-3.1-8B to denoise OCR errors and focus on prognostic details.
- Fusion: Individual Cox Proportional Hazards (CoxPH) models were trained for each modality. A second-stage CoxPH model then performed "late fusion" by taking the risk scores from unimodal models as its input.
Data and Preprocessing
The study utilized The Cancer Genome Atlas (TCGA), encompassing 7,982 patient cases across 32 cancer types. The pipeline involved extracting pathology report text and precomputed image embeddings, followed by Principal Component Analysis (PCA) to compress high-dimensional embeddings into manageable sizes (ranging from 4 to 256 dimensions).
Key Results and Insights
Multimodal Superiority
The primary finding is that multimodal fusion consistently outperforms unimodal models. The fusion model reached a peak average C-index of approximately 0.795, compared to unimodal peaks around 0.75. This performance gain was consistent across the most prevalent cancer types, including BRCA, KIRC, and LUAD.
The Clinical Baseline and Additive Information
A striking observation was the strength of the clinical baseline; models using just five tabular features (age, sex, race, etc.) performed remarkably well (C-index ~0.630). While complex embeddings from images or genomics did not always vastly outperform this baseline individually, they proved to be non-redundant. Combining them consistently improved accuracy, suggesting that FMs capture unique biological signals absent from clinical charts.
The Power of LLM Summarization
Pathology reports, often ignored due to noise, became a high-value modality when processed correctly. Using an LLM to summarize these reports boosted the C-index from 0.694 to 0.752. While manual review of 40 cases revealed minor LLM hallucinations, these errors were generally insufficient to alter the patient's overall survival risk score.
Conclusion
This work argues for a new era in biomedical modeling focused on the orchestration of foundation models. By using FMs as feature extractors and LLMs as intelligent denoisers, researchers can build powerful, interpretable survival models without the overhead of bespoke deep learning architectures. Future work will focus on validating this framework in "real-world" clinical settings and exploring the potential of fine-tuning embeddings specifically for survival tasks.
Summary updated