Precision oncology relies heavily on identifying specific molecular biomarkers—such as genetic mutations, gene expression profiles, or protein abundance—to tailor therapies to individual patients. For instance, determining whether a breast cancer tumor is HER2-positive dictates the use of targeted drugs like trastuzumab, while identifying Microsatellite Instability (MSI) or Homologous Recombination Deficiency (HRD) determines eligibility for immunotherapies or PARP inhibitors.
However, standard molecular diagnostic techniques (such as Next-Generation Sequencing, RNA sequencing, or specialized immunohistochemistry) are resource-intensive, expensive, and slow. In many community hospitals or resource-constrained healthcare settings, the infrastructure and expertise required for these tests are unavailable, creating a significant barrier to equitable patient care.
To bridge this gap, a rapidly growing research direction leverages deep learning to extract these critical molecular insights directly from standard, low-cost, and universally available diagnostic materials—most notably, Hematoxylin and Eosin (H&E) stained histopathology slides.
The Morphological Fingerprint: Why It Works
The fundamental premise of this approach is that a tumor's genotype directly influences its phenotype. Genetic mutations and altered transcriptional programs do not occur in a vacuum; they fundamentally remodel cellular architecture, alter nuclear shapes, change cell density, and modify the surrounding tumor microenvironment or stroma.
While these subtle structural changes are often invisible or unquantifiable to the human eye, deep learning models can detect this "morphological fingerprint" across millions of pixels in a whole-slide image (WSI). As demonstrated by Salim Arslan et al. in A systematic pan-cancer study on deep learning-based prediction of multi-omic biomarkers from routine pathology images, standard pathology slides contain rich, multi-omic signals reflecting a tumor's genetic, transcriptomic, and proteomic makeup.
To build intuition for why this mapping exists, Hanna M. Hieromnimon et al. developed a generative pipeline in Building digital histology models of transcriptional tumor programs with generative deep learning for pathology-based precision medicine. By using Generative Adversarial Networks (GANs) to translate complex gene expression patterns into synthetic tissue slides, the authors visually isolated the exact morphological features—such as stroma density and nuclear pleomorphism—that correspond to specific molecular programs. This confirms that deep learning models are not merely memorizing statistical noise, but are capturing genuine biological manifestations of molecular states.
Decoding Diverse Molecular Targets
Researchers have successfully trained models to predict a wide array of critical biomarkers across various cancer types:
- Receptor Status and Targetable Mutations: In breast cancer, Gil Shamai et al. showed in Clinical utility of receptor status prediction in breast cancer and misdiagnosis identification using deep learning on hematoxylin and eosin-stained slides that deep learning can accurately predict ER, PR, and ERBB2 (HER2) status directly from H&E slides. Similarly, Dmitrii Bychkov et al. in Deep learning identifies morphological features in breast cancer predictive of cancer ERBB2 status and trastuzumab treatment efficacy and Bao Li et al. in Point Transformer with Federated Learning for Predicting Breast Cancer HER2 Status from Hematoxylin and Eosin-Stained Whole Slide Images demonstrated robust HER2 status prediction. Beyond breast cancer, Benjamin Liechty et al. applied multi-scale deep learning ensembles to predict prognostic IDH mutations in infiltrating gliomas from standard slides in Machine learning can aid in prediction of IDH mutation from H&E-stained histology slides in infiltrating gliomas.
- DNA Repair and Immunotherapy Biomarkers: Predicting therapeutic response is another major frontier. Ke Zhang et al. (Development of model for identifying homologous recombination deficiency (HRD) status of ovarian cancer with deep learning on whole slide images) and Dan‐Yang Yuan et al. (Breast cancer homologous recombination deficiency prediction from pathological images with a sufficient and representative Transformer) successfully predicted HRD status, which is vital for identifying candidates for PARP inhibitors. For immunotherapy, Charlie Saillard et al. validated an AI pre-screening tool called MSIntuit to detect Microsatellite Instability (MSI) from colorectal cancer slides in Validation of MSIntuit as an AI-based pre-screening tool for MSI detection from colorectal cancer histology slides.
- Continuous Transcriptomic Signatures: Rather than treating biomarkers as binary classifications, some frameworks predict full continuous expression profiles. Areej Alsaafin et al. introduced the tRNAsformer in Learning to predict RNA sequence expressions from whole slide images with applications for search and classification to infer bulk RNA-seq expression levels directly from WSIs. Similarly, A. García demonstrated in Artificial intelligence-based pathology as a biomarker of sensitivity to atezolizumab–bevacizumab in patients with hepatocellular carcinoma: a multicentre retrospective study that a deep learning model could infer a complex 10-gene expression signature from standard liver biopsies to predict immunotherapy survival outcomes.
Algorithmic Mechanisms for Gigapixel Images
The technical challenge of processing histopathology slides stems from their sheer scale: a single whole-slide image can contain billions of pixels, making direct end-to-end deep learning computationally intractable.
To overcome this, researchers utilize advanced architectures like Multiple Instance Learning (MIL) and Vision Transformers. The image is typically broken down into thousands of smaller tissue patches. Models then aggregate features across these patches to make a slide-level prediction. For example, Dan‐Yang Yuan et al. utilized a specialized "SuRe-Transformer" to capture representative global context across patches for HRD prediction.
Alternatively, models can extract explicit "pathomic" texture features and cell density maps. Fabien Lamaze demonstrated in Computational analysis of whole slide images predicts PD-L1 expression and progression-free survival in immunotherapy-treated non-small cell lung cancer patients that modeling spatial cell density maps yields interpretable digital image patterns that correlate directly with biological immune activity and PD-L1 expression.
Extending to Multi-Modal and Clinical Data
While histopathology is the primary driver of this paradigm, the concept of extracting molecular insights from routine, non-invasive data extends to other modalities. For instance, Jiexiao Wang et al. developed a model using CT radiomics to predict Tumor Mutational Burden (TMB) and immunotherapy response in lung cancer in CT radiomics-based model for predicting TMB and immunotherapy response in non-small cell lung cancer.
To maximize predictive power, some frameworks combine multiple routine data streams. Quan Liu et al. proposed $M^2$Fusion in M2Fusion: Bayesian-based Multimodal Multi-level Fusion on Colorectal Cancer Microsatellite Instability Prediction, which fuses pathology WSIs with 3D radiology CT scans to significantly improve MSI prediction accuracy. Even purely clinical parameters can be leveraged: Konstantinos Rounis et al. showed in Prediction of outcome in patients with non-small cell lung cancer treated with second line PD-1/PDL-1 inhibitors based on clinical parameters: Results from a prospective, single institution study that routinely collected data like BMI, antibiotic use, and metastasis locations can be processed via an automated machine learning pipeline to predict immunotherapy response with high accuracy.
Clinical Workflow Integration: Pre-Screening and Triage
As reviewed by Didem Çifçi et al. in AI in Computational Pathology of Cancer: Improving Diagnostic Workflows and Clinical Outcomes?, the goal of these computational models is not necessarily to completely replace gold-standard genomic sequencing, but to optimize clinical workflows.
By acting as highly sensitive pre-screening or "rule-out" tools, these models can drastically reduce the volume of expensive confirmatory tests required. For example, the MSIntuit tool validated by Charlie Saillard et al. achieved near-perfect sensitivity, allowing clinicians to safely rule out MSI in nearly half of the colorectal cancer patient population using only standard H&E slides. This triage mechanism dramatically lowers healthcare costs, accelerates turnaround times, and ensures that limited molecular testing resources are directed precisely where they are needed most.
Key Foundational Work
Development and interpretation of a pathomics-based model for the prediction of microsatellite instability in Colorectal Cancer (2020) — contributed 1 concept elements to this direction.
Deep learning identifies morphological features in breast cancer predictive of cancer ERBB2 status and trastuzumab treatment efficacy (2021) — contributed 1 concept elements to this direction.
Prediction of outcome in patients with non-small cell lung cancer treated with second line PD-1/PDL-1 inhibitors based on clinical parameters: Results from a prospective, single institution study (2021) — contributed 1 concept elements to this direction.