TL;DR
This paper introduces IGI-DL, a novel deep learning model that predicts gene spatial expression directly from standard Hematoxylin and Eosin (HE) stained histology images. By integrating image texture and structural information from cell graphs, IGI-DL achieves superior gene expression prediction and enables more accurate cancer prognosis than existing methods, offering a more accessible alternative to expensive spatial transcriptomics.
What Problem Does This Paper Address?
Understanding the tumor microenvironment (TME) is crucial for cancer diagnosis and prognosis. Spatial transcriptomics provides rich data on gene expression and its location within tissues, offering a detailed view of the TME. However, the high cost and complex infrastructure required for spatial transcriptomics limit its widespread clinical application. This creates a need for more accessible methods to infer gene spatial expression and TME characteristics from readily available HE-stained histology images.
Methods
The core innovation is the Integrated Graph and Image Deep Learning (IGI-DL) model. It combines two key components:
- Image Branch: A ResNet18 convolutional neural network (CNN) extracts features related to tissue texture and morphology from HE image patches.
- Graph Branch: A Graph Isomorphism Network (GIN), a type of Graph Neural Network (GNN), processes Nuclei-Graphs. These graphs are constructed by segmenting nuclei within HE image patches and representing each nucleus as a node with extracted features. Edges connect nuclei based on their spatial proximity. This branch captures the structural and geometric relationships between cells.
These two branches are then fused, and the combined features are used to predict the spatial expression of target genes. For prognosis, IGI-DL is used to predict gene expression, which then serves as node features for a super-patch graph. This graph, along with clinical data, is fed into a graph-based survival model (using Graph Attention Networks - GAT) to generate a patient-level risk score.
Datasets
The study utilized several datasets:
- In-house CRC dataset: Six colorectal cancer patients' data for leave-one-patient-out validation.
- Public CRC external test set: One colorectal cancer patient's data.
- Public breast cancer validation set: Data from eight breast cancer patients (ST technology).
- Public breast cancer external test set: Data from one breast cancer patient (ST technology).
- Public cSCC validation set: Data from four cutaneous squamous cell carcinoma patients (ST technology).
- Public cSCC external test set: Data from one cSCC patient (10X Visium technology).
- NCT-CRC-HE-100K dataset: Used for latent space visualization.
- TCGA-CRC and TCGA-BRCA cohorts: Used for prognosis prediction, containing whole slide images (WSIs) and survival data.
Key Insights and Ideas
- Bridging the Gap: IGI-DL successfully bridges the gap between easily accessible HE images and the rich molecular information from spatial transcriptomics, making TME analysis more feasible.
- Synergy of Image and Graph: Combining texture features from CNNs with structural features from GNNs significantly outperforms models relying on either modality alone for gene expression prediction.
- Structural Information is Key: The graph-based component, capturing cell-cell relationships and tissue architecture, is crucial for accurate gene expression prediction, going beyond simple pixel intensity.
- Prognostic Power of Predicted Genes: Gene expression predicted from HE images alone can be used to construct a powerful prognostic model, demonstrating the clinical relevance of the IGI-DL predictions.
- Cross-Platform Generalization: The model shows good generalization ability across different spatial transcriptomics platforms (ST and 10X Visium), even when tested on data from different platforms than it was trained on.
Key Results
- Gene Expression Prediction: IGI-DL consistently outperformed three state-of-the-art (SOTA) models (ST-Net, HisToGene, Hist2ST) in predicting gene spatial expression across colorectal cancer (CRC), breast cancer, and cutaneous squamous cell carcinoma (cSCC).
- It enhanced the mean correlation of the top five predicted genes by an average of 0.125 across internal and external test sets, increasing it from 0.306 to 0.431.
- For CRC, the mean Pearson correlation reached 0.161 (leave-one-patient-out) and 0.146 (external test).
- For breast cancer, it achieved 0.200 (leave-one-patient-out) and 0.213 (external test).
- For cSCC, it reached 0.188 (leave-one-patient-out) and 0.191 (external test).
- Prognosis Prediction: The super-patch graph risk score derived from IGI-DL predicted gene expression demonstrated superior prognostic accuracy.
- For CRC, it achieved a C-index of 0.713.
- For breast cancer, it achieved a C-index of 0.741.
- These scores surpassed traditional HE-based risk scores.
- Latent Space Visualization: t-SNE and UMAP visualizations showed that IGI-DL can effectively distinguish between different tissue components (lymphocytes, stroma, tumor) based on predicted gene expression.
Limitations and Future Work
- Platform Specificity: The current model is designed for ST and 10X Visium platforms and has not been tested on other spatial transcriptomics technologies like MERFISH or Slide-seq.
- Resolution: The model does not achieve super-resolution prediction.
- Future Directions: The authors aim to extend gene spatial expression and prognosis prediction to more solid tumors. They also plan to explore leveraging histological images and spatial gene expression to infer evolutionary trajectories in tumor tissues for a deeper understanding of tumor onset and progression.
Summary updated