A comprehensive benchmarking for spatially resolved transcriptomics clustering methods across variable technologies, organs, and replicates

Renjie Chen, Yue Yao, Jingyang Qian, Xin Peng, Xin Shao, Xiaohui FaniMeta 2025 · DOI

TL;DR

This paper provides a comprehensive benchmark of 14 spatial transcriptomics clustering methods, revealing that no single method excels across all technologies, organs, or biological replicates. The study offers practical guidance for researchers to select the most appropriate method based on their specific data characteristics and experimental goals, moving beyond ad-hoc evaluations.

What Problem Does This Paper Address?

Spatially resolved transcriptomics (SRT) is revolutionizing our understanding of tissue heterogeneity by capturing gene expression within its spatial context. A crucial downstream analysis is spatial clustering, which identifies distinct tissue regions or domains. However, the rapid development of numerous computational tools for this task, coupled with the diversity of SRT technologies (e.g., sequencing-based vs. imaging-based), organs, and biological variability, has made it challenging to select the most suitable method. Previous benchmarks have often been limited in scope, raising questions about the generalizability of their findings. This paper addresses this critical need by providing a systematic and comprehensive evaluation of 14 leading spatial clustering methods across a wide array of real-world and simulated datasets, aiming to offer robust guidance for the SRT community.

Methods

The authors evaluated 14 spatial clustering methods, encompassing both traditional statistical approaches and newer neural network-based methods. Their benchmarking strategy involved:

  • Real-world datasets: Utilizing approximately 600 datasets from 10 different SRT technologies (e.g., ST, 10x Visium, Slide-seq, Stereo-seq, seqFISH+, MERFISH, CosMx, Xenium) across 8 organs. Ground truth for these datasets was derived from expert annotations or previously validated clustering results.
  • Simulated datasets: Employing simulators like scCube and SRTsim to generate controlled datasets. This allowed for the evaluation of method performance under specific conditions, including:
  • Biological replicates: Simulating variations between tissue slices from the same biological sample.
  • Spatial neighborhood disruptions: Introducing artificial changes to cluster boundaries and adjacency to test robustness.
  • Variable spatial patterns: Systematically altering data characteristics like spot number, gene number, sparsity, cluster number, and cluster shape.
  • Evaluation metrics: A suite of six widely adopted metrics (ARI, NMI, FMI, Purity, Homogeneity, Completeness) were used to quantify clustering accuracy, with an average rank across these metrics used to derive an overall accuracy score. Additional metrics like Average Silhouette Width (ASW), CHAOS, and Percentage of Abnormal Spots (PAS) were used to assess spatial continuity.
  • Computational efficiency: Runtime and peak memory usage were also recorded.
  • Preprocessing pipeline analysis: The impact of various preprocessing steps (normalization, log transformation, gene selection, standardization, dimension reduction) on clustering accuracy was systematically investigated.

Datasets

The study utilized a substantial collection of datasets:

  • Real-world datasets: Approximately 600 datasets from 10 different SRT technologies (ST, 10x Visium, Slide-seq, Stereo-seq, Visium HD, seqFISH+, STARmap, MERFISH, CosMx, Xenium) spanning 8 organs (brain, breast, heart, intestine, kidney, liver, lung, skin). These were sourced from public repositories like Cellxgene, STOmicsDB, Single Cell Portal, SODB, and SPASCER.
  • Simulated datasets: Generated using scCube and SRTsim to create controlled scenarios for evaluating biological replicates, spatial neighborhood disruptions, and variable spatial patterns.

The study did not introduce new benchmark datasets but rather leveraged existing public resources and advanced simulation tools to create a comprehensive evaluation framework.

Key Insights and Ideas

  • No "one-size-fits-all" method: The most striking insight is the lack of a universally superior spatial clustering method. Performance is highly context-dependent, emphasizing the need for careful method selection.
  • Sparsity is a major challenge: High sparsity in SRT data significantly degrades clustering accuracy for most methods, highlighting the importance of algorithms robust to sparse data.
  • Spatial continuity matters, but context is key: Methods that assume strong spatial continuity perform well in layered tissues but can struggle with more complex, irregular structures. Flexible spatial modeling is crucial for heterogeneous environments.
  • Preprocessing is critical: Default preprocessing pipelines are often suboptimal. Tailoring normalization, gene selection, and dimensionality reduction steps can substantially improve clustering robustness and accuracy.
  • Cell type vs. spatial domain identification: Different methods show distinct strengths for identifying broad spatial domains versus fine-grained cell types, especially with high-resolution technologies.

Key Results

  • Method performance varies by technology and organ: For instance, STAGATE and GraphST performed well on 10x Visium brain datasets, while BASS and BayesSpace were effective on ST datasets. Banksy showed optimal accuracy on CosMx datasets.
  • Organ-specific challenges: Clustering accuracy differed across organs, with layered structures (e.g., brain cortex) generally yielding better results than organs with irregular structures (e.g., liver).
  • Robustness to replicates and disruptions: While most methods showed comparable performance on simulated replicates, some, like DeepST, SEDR, and STAGATE, demonstrated greater resilience to spatial neighborhood disruptions.
  • Sparsity's impact: As sparsity increased, ARI scores generally declined, with BASS, Banksy, and DeepST showing the most robustness.
  • Optimal preprocessing pipeline: A general optimal pipeline was identified: library-size normalization, log transformation, selection of 3000 spatially variable genes, no standardization, and top 20 principal components for dimensionality reduction.

Limitations and Future Work

  • Limited scope of methods: The study benchmarked 14 methods, acknowledging that many other specialized tools exist.
  • Focus on accuracy: The primary focus was on clustering accuracy, not inferring causality between data features and algorithmic performance.
  • Single-cell vs. spatial: While spatial clustering is the focus, many methods also support other tasks like cell-type deconvolution and slice integration, which were not exhaustively evaluated.
  • Future directions: The authors suggest that integrating additional information, such as spatial multi-omics and H&E staining images, could offer further improvements. They also highlight the need for more optimized strategies for multimodal integration to fully leverage tissue morphology.

Summary updated

Abstract

Spatial clustering is a critical step in the analysis of spatially resolved transcriptomics, serving as the foundation for downstream investigation of tissue heterogeneity. Although numerous computational tools have been developed, systematic benchmarking across different technologies, organs, and biological replicates has been limited. Here, we present a comprehensive evaluation of 14 spatial clustering methods using approximately 600 datasets, including both real-world and simulated data with ground truth. We evaluated accuracy and applicability across diverse technologies and organs, revealing method-specific strengths and preferences. Using simulation of adjacent tissue slices and spatial neighborhood disruptions, we further examined performance in the context of biological replicates. Furthermore, we investigated how data characteristics, spatial distribution patterns, and preprocessing pipelines influence clustering outcomes. Together, our results provide practical benchmarking guidance, enabling researchers to select appropriate spatial clustering methods tailored to specific technologies, organs, and biological replicates.
BibTeX
@inproceedings{chen2025comprehensive,
  author = {Renjie Chen and Yue Yao and Jingyang Qian and Xin Peng and Xin Shao and Xiaohui Fan},
  title = {{A comprehensive benchmarking for spatially resolved transcriptomics clustering methods across variable technologies, organs, and replicates}},
  booktitle = {iMeta},
  year = {2025},
  doi = {10.1002/imt2.70084},
}
Figure