Automated cell annotation and classification on histopathology for spatial biomarker discovery

Zhe Li, Seyed Hossein Mirjahanmardi, Rasoul Sali, Feyisope Eweje, Matthew Gopaulchan, Leon Kloker, Xiaoming Zhang, Guoxin Li, Yuming Jiang, Ruijiang LiNature Communications 2025 · OpenReview

Abstract

Histopathology with hematoxylin and eosin (H&E) staining is the clinical standard for cancer diagnosis. However, manual single-cell annotation is inefficient and prone to human error, limiting the potential for complex spatial analysis. This study presents a computational framework that automates cell annotation by co-registering H&E images with multiplexed immunofluorescence (mIF). Using a dataset of 1,127,252 cells, the authors trained a deep learning model that achieves 86%–89% accuracy in classifying tumor cells, lymphocytes, neutrophils, and macrophages. Crucially, the model identifies spatial biomarkers, such as lymphocyte-neutrophil interactions, that predict patient survival and response to immunotherapy across multiple cancer types.

Introduction: The Annotation Crisis in Histopathology

In clinical oncology, Hematoxylin and Eosin (H&E) staining remains the gold standard for diagnosis. While these slides contain a wealth of information about the tumor microenvironment (TME), extracting single-cell data is notoriously difficult. Traditionally, researchers rely on pathologists to manually label cells—a process that is slow, prone to human error, and limited by human visual perception. For instance, while tumor cells and lymphocytes are relatively distinct, distinguishing macrophages from other cells on an H&E slide is so challenging that inter-pathologist agreement often hovers around 50%. This "annotation crisis" has historically prevented the development of robust AI models capable of complex spatial analysis.

Methodology: Bridging mIF and H&E

To overcome the limitations of manual labeling, the researchers developed a framework to generate high-quality labels without human intervention. The process began with serial staining: multiplexed immunofluorescence (mIF) was performed on a tissue section to identify cells via protein markers (e.g., CD3 for T-cells, pan-CK for tumor cells). The slide was then washed and stained with standard H&E.

A sophisticated alignment algorithm was used for single-cell co-registration, mapping mIF labels onto H&E images at the single-cell level. This effectively used mIF as a "ground truth" for training. The deep learning architecture employed a ResNet18 backbone enhanced by two key strategies:

  1. Self-Supervised Learning (SSL): Pre-training on unlabeled cells from the CPTAC dataset using a Bootstrap Your Own Latent (BYOL) approach to learn general morphological features.
  2. Domain Adaptation: Utilizing a Gradient Reversal Layer (GRL) to ensure model robustness across different medical centers and staining variations, which often degrade medical AI performance.
The deep learning architecture integrates self-supervised learning for feature extraction and domain adaptation to ensure cross-site robustness.

Results: High-Accuracy Classification and Prognostic Power

The model achieved an overall accuracy of 86%–89% across internal and external validation sets, including 3,263 whole-slide images from The Cancer Genome Atlas (TCGA). Notably, models trained on these automated mIF labels significantly outperformed those trained on manual human annotations, which reached only ~56% accuracy in similar tasks. This suggests that H&E images contain subtle morphological signatures that distinguish cell types like macrophages and neutrophils, even if they are inconsistent to the human eye.

Beyond classification, the model enabled large-scale spatial analysis. By generating cell identity maps, the researchers used point-pattern analysis to measure how different cell types interact spatially.

A representative cell identity map showing the spatial distribution of classified cell types within the tumor microenvironment.

The study revealed that the proximity of specific cells is a powerful biomarker. Specifically, the "lymphocyte-neutrophil interaction" was identified as a strong independent predictor of survival across 10 different patient cohorts (Hazard Ratios ranging from 1.91 to 5.75). In patients treated with immune checkpoint inhibitors, this spatial biomarker achieved an AUC of 0.817–0.832 for predicting treatment response, doubling the sensitivity of the current clinical standard (PD-L1 CPS) at 90% specificity.

Discussion and Future Directions

While the current model focuses on four major cell types (tumor cells, lymphocytes, neutrophils, and macrophages), it demonstrates the feasibility of extracting high-dimensional insights from standard, low-cost H&E slides. Future work may involve using higher-plex imaging (such as CODEX) to train models to identify more specific cell states, such as exhausted versus active T-cells. By bypassing the need for manual annotation, this scalable approach paves the way for the discovery of novel spatial biomarkers in precision oncology.

Summary updated

Abstract

Histopathology with hematoxylin and eosin (H&E) staining is routinely employed for clinical diagnoses. Single-cell analysis of histopathology provides a powerful tool for understanding the intricate cellular interactions underlying disease progression and therapeutic response. However, existing efforts are hampered by inefficient and error-prone human annotations. Here, we present an experimental and computational approach for automated cell annotation and classification on H&E-stained images. Instead of human annotations, we use multiplexed immunofluorescence (mIF) to define cell types based on cell lineage protein markers. By co-registering H&E images with mIF of the same tissue section at the single-cell level, we create a dataset of 1,127,252 cells with high-quality annotations on tissue microarray cores. A deep learning model combining self-supervised learning with domain adaptation is trained to classify four cell types on H&E images with an overall accuracy of 86%-89%, and the cell classification model is applicable to whole slide images. Further, we show that spatial interactions among specific immune cells in the tumor microenvironment are linked to patient survival and response to immune checkpoint inhibitors. Our work provides a scalable approach for single-cell analysis of standard histopathology and may enable discovery of novel spatial biomarkers for precision oncology. Histopathology is essential for diagnosing cancer, but cell annotation remains unreliable and inefficient. Here, authors develop a method using deep learning and immune-fluorescence-guided cell labelling that enables accurate cell identification and links spatial cellular patterns to patient outcomes.
BibTeX
@inproceedings{li2025automated,
  author = {Zhe Li and Seyed Hossein Mirjahanmardi and Rasoul Sali and Feyisope Eweje and Matthew Gopaulchan and Leon Kloker and Xiaoming Zhang and Guoxin Li and Yuming Jiang and Ruijiang Li},
  title = {{Automated cell annotation and classification on histopathology for spatial biomarker discovery}},
  booktitle = {Nature Communications},
  year = {2025},
  url = {https://openreview.net/forum?id=65JZb6q9sv},
}
Figure