Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images

Richard J. Chen, Ming Y. Lu, Wei-Hung Weng, Tiffany Y. Chen, Drew F. K. Williamson, Trevor Manz, Maha Shady, Faisal Mahmood

Introduction

Predicting patient survival from cancer pathology is a significant challenge due to the sheer scale of data. Digital Whole Slide Images (WSIs) often exceed 100,000 pixels in each dimension, creating a "needle-in-a-haystack" scenario for identifying prognostic features. While genomic data provides critical biological context, traditional multimodal approaches often fail to capture the intricate spatial relationships between a patient's genetic profile and the tumor microenvironment.

The Multimodal Co-Attention Transformer (MCAT) addresses these challenges by treating genomic data as a guide for visual feature extraction. Rather than simply concatenating modalities, MCAT uses genomic features to "attend" to relevant histology patches, effectively compressing massive images into biologically meaningful representations.

Methodology

The MCAT framework processes both WSIs and genomic data as sets of features, employing a novel co-attention mechanism to bridge the gap between modalities.

Overview of the Multimodal Co-Attention Transformer (MCAT) architecture.

Genomic Bagging and Co-Attention

Instead of treating genomic data as a flat vector, MCAT groups genes into $N$ functional categories (e.g., Oncogenesis, Protein Kinases). These categories are embedded into a latent space to form a "genomic bag."

The core innovation is the Genomic-Guided Co-Attention (GCA) layer. In this layer, the $N$ genomic embeddings serve as Queries that attend to thousands of histology patch embeddings, which act as Keys and Values. This mechanism serves as a dimensionality bottleneck, compressing the massive WSI into a fixed-size set of "gene-guided visual concepts."

Set-based MIL Transformers

Following compression, the model utilizes standard Transformer layers to model global interactions between these $N$ concepts. This approach avoids the quadratic computational cost associated with self-attention on the original sequence of 100,000+ image patches, enabling efficient end-to-end training.

Experimental Setup

The model was rigorously evaluated using 4,370 WSIs from five large-scale datasets in The Cancer Genome Atlas (TCGA):

  • BLCA: Bladder Urothelial Carcinoma
  • BRCA: Breast Invasive Carcinoma
  • GBMLGG: Glioblastoma & Lower Grade Glioma
  • LUAD: Lung Adenocarcinoma
  • UCEC: Uterine Corpus Endometrial Carcinoma

The total dataset comprised approximately 67 million image patches, providing a robust benchmark for survival prediction performance.

Results and Discussion

MCAT consistently outperformed state-of-the-art survival prediction methods, including Attention MIL and DeepAttnMISL. The model achieved an overall concordance index (c-Index) of 0.653, marking a 3.0% to 6.87% improvement over existing multimodal fusion strategies.

Visual interpretation of genomic-guided co-attention.

Beyond quantitative gains, MCAT offers high interpretability. By visualizing the co-attention weights, researchers can generate heatmaps that link specific biological pathways to tissue morphology. For instance, in breast cancer cases, the model's "Oncogenesis" queries were found to attend specifically to regions of high tumor cellularity, while "Cytokine" queries highlighted immune cell infiltrates. This "genomics as a search engine" approach allows for the discovery of spatially resolved biomarkers.

Conclusion and Future Work

The MCAT framework represents a significant step forward in multimodal cancer prognosis. By using genomics to guide visual attention, it overcomes the computational hurdles of gigapixel images while maintaining biological interpretability.

Future research may focus on refining the genomic categories, which are currently manually curated and may contain overlapping functions. Additionally, replacing the ImageNet-pretrained ResNet-50 backbone with a pathology-specific self-supervised model could further enhance the system's predictive power. Expanding the framework to include other "omics" modalities, such as spatial transcriptomics, remains a promising avenue for future exploration.

Summary updated

Abstract

Survival outcome prediction is a challenging weakly-supervised and ordinal regression task in computational pathology that involves modeling complex interactions within the tumor microenvironment in gigapixel whole slide images (WSIs). Despite recent progress in formulating WSIs as bags for multiple instance learning (MIL), representation learning of entire WSIs remains an open and challenging problem, especially in overcoming: 1) the computational complexity of feature aggregation in large bags, and 2) the data heterogeneity gap in incorporating biological priors such as genomic measurements. In this work, we present a Multimodal Co-Attention Transformer (MCAT) framework that learns an interpretable, dense co-attention mapping between WSIs and genomic features formulated in an embedding space. Inspired by approaches in Visual Question Answering (VQA) that can attribute how word embed-dings attend to salient objects in an image when answering a question, MCAT learns how histology patches attend to genes when predicting patient survival. In addition to visualizing multimodal interactions, our co-attention trans-formation also reduces the space complexity of WSI bags, which enables the adaptation of Transformer layers as a general encoder backbone in MIL. We apply our proposed method on five different cancer datasets (4,730 WSIs, 67 million patches). Our experimental results demonstrate that the proposed method consistently achieves superior performance compared to the state-of-the-art methods.
BibTeX
@inproceedings{chen2021multimodal,
  author = {Richard J. Chen and Ming Y. Lu and Wei-Hung Weng and Tiffany Y. Chen and Drew F. K. Williamson and Trevor Manz and Maha Shady and Faisal Mahmood},
  title = {{Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00398},
}
Figure