Introduction
Single-cell RNA sequencing (scRNA-seq) has transformed our understanding of biological systems by enabling the profiling of gene expression at the level of individual cells. A fundamental step in scRNA-seq analysis is clustering, which groups cells into distinct types or states based on their transcriptomic profiles. However, the rapid proliferation of clustering algorithms—ranging from traditional statistical methods to advanced deep learning and graph-based models—has created a fragmented landscape. Researchers often lack standardized benchmarks to determine which tools are most effective for their specific biological contexts, particularly when dealing with the high sparsity and noise inherent in single-cell data.
The scCluBench Framework
To address these challenges, we introduce scCluBench, a standardized and comprehensive benchmarking framework designed to evaluate the performance of clustering algorithms across a wide range of scenarios. The framework is built on four pillars of standardization: curated benchmark resources, rigorous evaluation protocols, biological interpretation pipelines, and a unified modular workflow.
scCluBench evaluates 14 diverse models categorized into four groups:
- Traditional: Louvain, Leiden, and SC3.
- Deep Learning-based: scMAE, DEC, DESC, scDeepCluster, scNAME, scDCC, and scziDesk.
- Graph-based: scCDCG, scGNN, and scDSC.
- Foundation Models: scGPT and GeneFormer.
Dataset Diversity and Complexity
The benchmark utilizes 36 gold-standard scRNA-seq datasets covering 18 tissue types across human and mouse specimens. These datasets were selected to represent real-world complexity, including varying scales and high levels of missing data points.
Key characteristics of the benchmark datasets include:
- Scale: Datasets range from small samples to "stress-test" sets with over 20,000 cells and 60,000 genes.
- Sparsity: 34 of the 36 datasets exhibit sparsity rates exceeding 80%, reflecting the technical noise common in scRNA-seq.
- Biological Breadth: Tissues include brain, lung, liver, pancreas, and more, ensuring the benchmark's relevance across diverse biological studies.
Key Results and Insights
Our comprehensive evaluation revealed several critical insights into the current state of scRNA-seq clustering:
- Top Performers: The graph-based model scCDCG emerged as the overall top performer, demonstrating exceptional ability to maintain clear cluster boundaries and handle complex data structures. Among deep learning methods, scMAE (a masked autoencoder) proved to be the most robust, particularly in high-sparsity environments.
- Foundation Model Performance: While biological foundation models like scGPT are powerful for general representation, they often underperform on task-specific clustering compared to specialized architectures. This suggests that "one-size-fits-all" models may still require task-specific optimization.
- The GNN Trap: Many Graph Neural Network (GNN) models suffer from "embedding collapse," where different cell types become indistinguishable. scCDCG avoids this through a unique "cut-informed" mechanism.
- Biological Prior Knowledge: Incorporating biological information, such as known marker genes, significantly enhances clustering accuracy. For instance, the DESC model saw a 28.1% improvement in accuracy when utilizing marker-overlap annotation compared to standard statistical mapping.
Conclusion
scCluBench provides a rigorous roadmap for the single-cell community, highlighting the strengths and weaknesses of current clustering technologies. While AI-driven approaches like scCDCG and scMAE are beginning to surpass traditional methods, challenges remain in scalability and stability. Future work will focus on expanding the framework to include multi-modal data and spatial transcriptomics, ensuring scCluBench remains at the forefront of single-cell analysis.
Summary updated