Leveraging Spatial Transcriptomics as Alternative to Manual Annotations for Deep Learning-Based Nuclei Analysis

Kazuya Nishimura, Ryoma Bise, Haruka Hirose, Yasuhiro KojimaarXiv 2026 · arXiv

TL;DR

This paper introduces a novel framework that uses spatial transcriptomics (ST) data to automatically generate training labels for deep learning-based nuclei segmentation and classification in pathology images. By extracting objective nuclear masks from DAPI stains and inferring cell types from gene expression profiles, the method eliminates the need for tedious manual pixel-level annotations while achieving superior generalization on unseen organs.

What Problem Does This Paper Address?

Deep learning models excel at segmenting and classifying cell nuclei in histopathology images, a task critical for quantitative precision medicine and cancer diagnostics. However, training these models requires massive amounts of pixel-level manual annotations.

Generating these labels is a notorious bottleneck: it is incredibly labor-intensive, expensive, and highly prone to inter-observer variability. Furthermore, models trained on one type of tissue often fail to generalize to diverse organs or varying staining conditions. The authors tackle this challenge by asking: Can we leverage automated molecular measurements to train image-based AI models without relying on human annotators?

Methods

The authors propose a cross-modal weakly supervised learning framework that utilizes data from emerging spatial transcriptomics (ST) platforms (such as 10x Genomics Xenium). These platforms simultaneously capture high-resolution H&E images, DAPI-stained nuclear images, and spatially resolved gene expression measurements from the exact same tissue section.

Fig. 1: Overview of proposed framework.

The pipeline consists of two main steps:

  1. Dataset Generation:
  2. Segmentation Labels: Nuclear boundary masks are automatically derived from DAPI staining, providing dense, high-quality structural supervision without human intervention.
  3. Classification Labels: Cell-type labels are inferred from gene expression profiles using a three-stage data-driven pipeline. Cells are first clustered using the Leiden algorithm. Cluster marker genes are then matched against reference databases (PanglaoDB, CellMarker) to assign coarse categories (Epithelial, Inflammatory, Connective). Finally, a neoplastic refinement step evaluates epithelial clusters against cancer-associated gene sets to identify malignant cells. Crucially, an explicit Unknown category is introduced to handle noisy or ambiguous molecular data.
  4. Model Training: A joint segmentation and classification model (CellViT architecture) is trained on the image patches. Pixels labeled as "Unknown" are simply excluded from the classification loss computation to mitigate label noise.

Datasets

The study utilizes three primary datasets:

  • HEST1K: A spatial transcriptomics dataset containing whole-slide H&E images, DAPI-derived masks, and gene expression profiles across 9 tissue types. It was used for model training and evaluating segmentation generalization.
  • PanNuke: A standard, fully manually annotated multi-organ dataset used as a baseline and validation benchmark.
  • Cls.-Eval: A separate, manually annotated breast cancer Xenium dataset from 10x Genomics used exclusively to evaluate cell-type classification performance.

Key Insights and Ideas

  • DAPI as a Super-Annotator: While human annotators get fatigued and frequently miss cells, DAPI stains provide dense, objective, and spatially aligned nuclear boundaries. This massive increase in training instances per slide drastically improves the model's robustness.
  • Bridging Molecules and Morphology: Standard transcriptomic cell-typing methods are designed for molecular analysis, not image recognition. By mapping fine-grained gene expression profiles into coarse, clinically relevant pathology categories—and adding an "Unknown" safety valve—the authors successfully bridge molecular biology and computer vision.

Key Results

  • Superior Segmentation Generalization: Despite being trained on fewer organ types (9 vs. 19), the model trained on ST data outperformed the fully supervised PanNuke baseline when evaluated on completely unseen organs. It achieved a higher Dice score (0.6585 vs. 0.6350) and a significant boost in Recall (0.8183 vs. 0.7862), proving it is much more robust at detecting true nuclei in unfamiliar tissues.
Fig. 3: Example of segmentation results.
  • Highly Competitive Classification: In cell-type classification experiments, the proposed hierarchical refinement approach achieved a Macro F1 score of 0.3492 and an Accuracy of 0.5359. This vastly outperformed existing reference-free transcriptomic labeling methods (such as UCell or CellAssign) and closely approached the performance of a model trained on fully manual annotations (PanNuke Accuracy: 0.5696).

Limitations and Future Work

The authors note that automated gene expression-based labeling still introduces a degree of intrinsic noise, which keeps the classification accuracy slightly below that of fully manual expert supervision.

As a next step, they suggest that incorporating a small amount of manually annotated data alongside this scalable, ST-derived weak supervision could further stabilize training and bridge the remaining performance gap.

Summary updated

Abstract

Deep learning-based nuclei segmentation and classification in pathology images typically rely on large-scale pixel-level manual annotations, which are costly and difficult to obtain across diverse tissues and staining conditions. To address this limitation, we propose a framework that leverages spatial transcriptomics (ST) data as supervision for nuclei segmentation and classification. By incorporating cell-level ST data, we obtain gene expression profiles and corresponding nuclear masks from histopathological images. Gene expression profiles are converted into cell-type labels and used as training data for image-based classification. Because existing gene expression-based cell-type classification methods are not designed for image recognition, we introduce an image-oriented classification approach that bridges gene expression-based cell typing and image-based cell classification. To evaluate generalization, we conduct segmentation experiments on previously unseen organs and compare our method with conventional supervised models. Despite being trained on fewer organ types, our framework achieves higher segmentation accuracy, demonstrating strong transferability. Classification experiments further show consistent improvements over existing approaches.
BibTeX
@misc{nishimura2026leveraging,
  author = {Kazuya Nishimura and Ryoma Bise and Haruka Hirose and Yasuhiro Kojima},
  title = {{Leveraging Spatial Transcriptomics as Alternative to Manual Annotations for Deep Learning-Based Nuclei Analysis}},
  year = {2026},
  eprint = {2604.23481},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.23481},
}

Loading related content...

Figure