Graph Attention-Based Fusion of Pathology Images and Gene Expression for Prediction of Cancer Survival

Yi Zheng, Regan D. Conrad, Emily J. Green, Eric J. Burks, Margrit Betke, Jennifer E. Beane, Vijaya B. KolachalamaIEEE TMI 2024 · DOI

TL;DR

This paper introduces a novel graph attention-based framework that fuses pathology images (WSIs) and gene expression data to predict patient survival in non-small cell lung cancer (NSCLC). By representing WSIs as graphs and employing an attention mechanism to integrate multimodal information, the model achieves state-of-the-art performance and offers interpretable "survival activation maps" that highlight prognostically relevant regions within tumors.

What Problem Does This Paper Address?

Predicting cancer survival is crucial for personalized treatment and prognosis. While pathology images (WSIs) and gene expression data offer complementary insights, effectively integrating these diverse modalities remains a challenge. Existing multimodal fusion methods often fail to fully capture the complex interactions between image-based spatial information and gene expression patterns. This paper aims to bridge this gap by developing a framework that can learn from both modalities simultaneously, leveraging the spatial context of WSIs and the molecular information from gene expression to improve survival prediction and provide interpretable insights into tumor biology.

Methods

The core of this work is a Graph Attention-Based Fusion Framework. Here's a breakdown of the key components:

  • WSI Processing and Graph Construction: Whole slide images are first processed through foreground-background separation and then tessellated into patches. These patches form the nodes of an undirected graph, with edges representing patch adjacency.
  • Patch Embedding: Each image patch is encoded into a feature vector using a Convolutional Neural Network (CNN) trained with contrastive learning. This creates the node features for the graph.
  • Graph Mixer (GML): This novel module, inspired by MLP-Mixer but adapted for graph data, consists of:
  • Node Mixing Layer (NML): Utilizes a Graph Attention Network (GAT) to learn local spatial relationships by allowing nodes to attend to their neighbors.
  • Channel Mixing Layer (CML): Mixes information across channels to capture higher-level feature interactions.
  • Genomic Signature Embeddings: Gene expression data is processed to create embeddings for specific gene signatures known to be associated with cancer phenotypes.
  • Genomic and Image Fusion (GAM): A Query-Key-Value (QKV) mechanism, specifically a Genomic Attention Module, is used to capture interpretable interactions between WSI patch embeddings and gene signature embeddings.
  • Global Attention Pooling: A gating-based weighted average of node features is computed to obtain a WSI-level representation.
  • Survival Prediction: A fully-connected layer takes the pooled representation and predicts patient survival risk.
  • Survival Loss Function: A cross-entropy-based Cox proportional loss function is employed for survival analysis.
  • Survival Activation Maps (SAM): To interpret the model's predictions, the authors adapted GradCAM to generate saliency maps that highlight regions in the WSI most associated with survival outcomes.

Datasets

The study utilized data from the following publicly available sources:

  • The Cancer Genome Atlas (TCGA): Used for training and validation.
  • Clinical Proteomic Tumor Analysis Consortium (CPTAC): Used as an independent test dataset.
  • National Lung Screening Trial (NLST): Used for fine-tuning feature generation.

The study focused on non-small cell lung cancer (NSCLC), specifically lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC).

Key Insights and Ideas

  • Graph Representation of WSIs: Treating WSIs as graphs allows the model to explicitly learn spatial relationships between image patches, which is crucial for understanding tissue microenvironments.
  • Attention-Based Multimodal Fusion: The attention mechanism is key to effectively integrating information from WSIs and gene expression, allowing the model to focus on the most relevant features from each modality.
  • Interpretability via SAMs: The development of Survival Activation Maps (SAMs) is a significant contribution, providing visual explanations for survival predictions and highlighting specific tumor regions and their association with prognosis. This moves beyond black-box models.
  • Synergy of Modalities: The fusion survival model (FSM) consistently outperforms the WSI-only model (ISM), demonstrating the added value of integrating gene expression data for improved survival prediction.
  • Robustness to Data Heterogeneity: The framework's ability to handle varying graph sizes (due to different WSI patch counts) is a practical advantage for real-world applications.

Key Results

  • State-of-the-Art Performance: The proposed Fusion Survival Model (FSM) achieved state-of-the-art performance in predicting NSCLC survival on both TCGA and CPTAC datasets, outperforming several other multimodal and unimodal methods.
  • Improved Concordance Index (c-index): For LUAD on TCGA, the FSM achieved a c-index of 0.703, and for LUSC on TCGA, it was 0.664.
  • Improved Time-Dependent Area Under the Curve (tAUC): The FSM achieved a tAUC of 0.679 for LUAD on TCGA and 0.792 for LUSC on CPTAC.
  • WSI-Only Model Performance: The Imaging Survival Model (ISM) also showed strong performance, outperforming many existing WSI-only methods, with a c-index of 0.687 for LUAD and 0.652 for LUSC on TCGA.
  • Qualitative Interpretability: SAMs generated by the FSM model showed a high degree of overlap with expert annotations and highlighted prognostically relevant histologic patterns and features, including vascular invasion and immune infiltrate.

Limitations and Future Work

  • Limited Expert Annotations: The study acknowledges that the comparison of SAMs with expert-driven annotations was performed on a small set of cases due to the tedious nature of manual annotation. A larger study with more comprehensive annotations is needed.
  • Negative Log-Likelihood (NLL) Loss Assumption: The NLL loss function assumes proportional hazards, which may not always hold true. This could lead to biased estimates if hazard ratios vary significantly over time.
  • Censoring Bias: Survival analysis is complicated by censored observations, which provide incomplete information. The authors suggest future work could address this by using inverse probability of censoring weighting.
  • Extension to Other Cancers and Omics: The authors propose extending the framework to other cancer types and integrating additional omics data (e.g., spatial transcriptomics, proteomics) for a broader understanding of cancer biology and survival.
  • Testing with Diverse Signatures: Future work will involve testing the model with a wider range of cell type and prognostic gene expression signatures.

Summary updated

BibTeX
@article{zheng2024graph,
  author = {Yi Zheng and Regan D. Conrad and Emily J. Green and Eric J. Burks and Margrit Betke and Jennifer E. Beane and Vijaya B. Kolachalama},
  title = {{Graph Attention-Based Fusion of Pathology Images and Gene Expression for Prediction of Cancer Survival}},
  journal = {IEEE TMI},
  year = {2024},
  doi = {10.1109/tmi.2024.3386108},
}
Figure