Predicting Gene Spatial Expression and Cancer Prognosis: An Integrated Graph and Image Deep Learning Approach Based on HE Slides

Ruitian Gao, Xin Yuan, Yanran Ma, Ting Wei, Luke Johnston, Yanfei Shao, Wenwen Lv, Tengteng Zhu, Yue Zhang, Junke Zheng, Guoqiang Chen, Jing Sun, Yu Guang Wang, Zhangsheng YubioRxiv 2023 · DOI

TL;DR

This paper introduces IGI-DL, a novel deep learning model that predicts gene spatial expression directly from standard Hematoxylin and Eosin (HE) stained histology images. By integrating image texture and structural information from cell graphs, IGI-DL achieves superior gene expression prediction and enables more accurate cancer prognosis than existing methods, offering a more accessible alternative to expensive spatial transcriptomics.

What Problem Does This Paper Address?

Understanding the tumor microenvironment (TME) is crucial for cancer diagnosis and prognosis. Spatial transcriptomics provides rich data on gene expression and its location within tissues, offering a detailed view of the TME. However, the high cost and complex infrastructure required for spatial transcriptomics limit its widespread clinical application. This creates a need for more accessible methods to infer gene spatial expression and TME characteristics from readily available HE-stained histology images.

Methods

The core innovation is the Integrated Graph and Image Deep Learning (IGI-DL) model. It combines two key components:

  • Image Branch: A ResNet18 convolutional neural network (CNN) extracts features related to tissue texture and morphology from HE image patches.
  • Graph Branch: A Graph Isomorphism Network (GIN), a type of Graph Neural Network (GNN), processes Nuclei-Graphs. These graphs are constructed by segmenting nuclei within HE image patches and representing each nucleus as a node with extracted features. Edges connect nuclei based on their spatial proximity. This branch captures the structural and geometric relationships between cells.

These two branches are then fused, and the combined features are used to predict the spatial expression of target genes. For prognosis, IGI-DL is used to predict gene expression, which then serves as node features for a super-patch graph. This graph, along with clinical data, is fed into a graph-based survival model (using Graph Attention Networks - GAT) to generate a patient-level risk score.

Datasets

The study utilized several datasets:

  • In-house CRC dataset: Six colorectal cancer patients' data for leave-one-patient-out validation.
  • Public CRC external test set: One colorectal cancer patient's data.
  • Public breast cancer validation set: Data from eight breast cancer patients (ST technology).
  • Public breast cancer external test set: Data from one breast cancer patient (ST technology).
  • Public cSCC validation set: Data from four cutaneous squamous cell carcinoma patients (ST technology).
  • Public cSCC external test set: Data from one cSCC patient (10X Visium technology).
  • NCT-CRC-HE-100K dataset: Used for latent space visualization.
  • TCGA-CRC and TCGA-BRCA cohorts: Used for prognosis prediction, containing whole slide images (WSIs) and survival data.

Key Insights and Ideas

  • Bridging the Gap: IGI-DL successfully bridges the gap between easily accessible HE images and the rich molecular information from spatial transcriptomics, making TME analysis more feasible.
  • Synergy of Image and Graph: Combining texture features from CNNs with structural features from GNNs significantly outperforms models relying on either modality alone for gene expression prediction.
  • Structural Information is Key: The graph-based component, capturing cell-cell relationships and tissue architecture, is crucial for accurate gene expression prediction, going beyond simple pixel intensity.
  • Prognostic Power of Predicted Genes: Gene expression predicted from HE images alone can be used to construct a powerful prognostic model, demonstrating the clinical relevance of the IGI-DL predictions.
  • Cross-Platform Generalization: The model shows good generalization ability across different spatial transcriptomics platforms (ST and 10X Visium), even when tested on data from different platforms than it was trained on.

Key Results

  • Gene Expression Prediction: IGI-DL consistently outperformed three state-of-the-art (SOTA) models (ST-Net, HisToGene, Hist2ST) in predicting gene spatial expression across colorectal cancer (CRC), breast cancer, and cutaneous squamous cell carcinoma (cSCC).
  • It enhanced the mean correlation of the top five predicted genes by an average of 0.125 across internal and external test sets, increasing it from 0.306 to 0.431.
  • For CRC, the mean Pearson correlation reached 0.161 (leave-one-patient-out) and 0.146 (external test).
  • For breast cancer, it achieved 0.200 (leave-one-patient-out) and 0.213 (external test).
  • For cSCC, it reached 0.188 (leave-one-patient-out) and 0.191 (external test).
  • Prognosis Prediction: The super-patch graph risk score derived from IGI-DL predicted gene expression demonstrated superior prognostic accuracy.
  • For CRC, it achieved a C-index of 0.713.
  • For breast cancer, it achieved a C-index of 0.741.
  • These scores surpassed traditional HE-based risk scores.
  • Latent Space Visualization: t-SNE and UMAP visualizations showed that IGI-DL can effectively distinguish between different tissue components (lymphocytes, stroma, tumor) based on predicted gene expression.

Limitations and Future Work

  • Platform Specificity: The current model is designed for ST and 10X Visium platforms and has not been tested on other spatial transcriptomics technologies like MERFISH or Slide-seq.
  • Resolution: The model does not achieve super-resolution prediction.
  • Future Directions: The authors aim to extend gene spatial expression and prognosis prediction to more solid tumors. They also plan to explore leveraging histological images and spatial gene expression to infer evolutionary trajectories in tumor tissues for a deeper understanding of tumor onset and progression.

Summary updated

Abstract

ABSTRACT Interpreting the tumor microenvironment (TME) heterogeneity within solid tumors presents a cornerstone for precise disease diagnosis and prognosis. However, while spatial transcriptomics offers a wealth of data, ranging from gene expression and spatial location to corresponding Hematoxylin and Eosin (HE) images, to explore the TME of various cancers, its high cost and demanding infrastructural needs significantly limit its clinical application, highlighting the need for more accessible alternatives. To bridge this gap, we introduce the Integrated Graph and Image Deep Learning (IGI-DL) model. This innovation, a fusion of Convolutional Neural Networks and Graph Neural Networks, is designed to predict gene spatial expression using HE images. The IGI-DL model outperforms its predecessors in analyzing colorectal cancer (CRC), breast cancer, and cutaneous squamous cell carcinoma (cSCC) by leveraging both pixel intensity and structural features in images. Significantly, across all cancer types, the IGI-DL model enhances the mean correlation of the top five genes by an average of 0.125 in internal and external test sets, rising from 0.306 to 0.431, surpassing existing state-of-the-art (SOTA) models. We further present a novel risk score derived from a super-patch graph, where gene expression predicted by IGI-DL serves as node features. Demonstrating superior prognostic accuracy, this risk score, with a C-index of 0.713 and 0.741 for CRC and breast cancer, supersedes traditional HE-based risk scores. In summary, the approach augments our understanding of the TME from the aspect of histological images, portending a transformation in cancer prognostics and treatment planning and ushering in a new era of personalized and precision oncology.
BibTeX
@misc{gao2023predicting,
  author = {Ruitian Gao and Xin Yuan and Yanran Ma and Ting Wei and Luke Johnston and Yanfei Shao and Wenwen Lv and Tengteng Zhu and Yue Zhang and Junke Zheng and Guoqiang Chen and Jing Sun and Yu Guang Wang and Zhangsheng Yu},
  title = {{Predicting Gene Spatial Expression and Cancer Prognosis: An Integrated Graph and Image Deep Learning Approach Based on HE Slides}},
  year = {2023},
  doi = {10.1101/2023.07.20.549824},
  howpublished = {bioRxiv},
}
Figure