Biology-driven insights into the power of single-cell foundation models

Jialu Wu, Qing Ye, Yilin Wang, Renling Hu, Yiheng Zhu, Mingze Yin, Tianyue Wang, Jike Wang, Chang-Yu Hsieh, Tingjun HouGenome 2025 · DOI

Abstract

This benchmark introduces novel evaluation perspectives, identifying the strengths and limitations of current scFMs, and paves the way for their effective application in biological and clinical research, including cell atlas construction, tumor microenvironment studies, and treatment decision-making.

Introduction

Single-cell RNA sequencing (scRNA-seq) has revolutionized biological research by enabling the observation of gene expression at individual cell resolution. However, the resulting data is often sparse, noisy, and high-dimensional. Inspired by the success of Large Language Models (LLMs), researchers have developed single-cell foundation models (scFMs)—large-scale neural networks pretrained on millions of cells to capture the underlying "language" of biology. While the potential of these models is vast, their actual performance in realistic biological and clinical scenarios has remained under-evaluated. This study presents a rigorous, "biology-first" benchmark of six prominent scFMs to determine if they truly offer superior insights compared to traditional methods.

Methods: A Zero-Shot Evaluation Protocol

The study evaluates six leading scFMs: Geneformer, scGPT, UCE, scFoundation, LangCell, and scCello. To assess the intrinsic biological knowledge captured during pretraining, the authors employed a zero-shot protocol. In this setup, the pretrained models are frozen and used as feature extractors to generate cell embeddings for downstream tasks. This approach ensures that the evaluation reflects the quality of the learned representations rather than the capacity of the model to be fine-tuned on specific task data.

Overview of the scFM architecture and embedding extraction process.

The Biology-First Evaluation Framework

A critical contribution of this work is the shift from purely technical metrics to ontology-informed metrics that reflect biological truth. These include:

  • scGraph-OntoRWR: Measures how well the model's latent space aligns with the expert-defined Cell Ontology graph.
  • Lowest Common Ancestor Distance (LCAD): Quantifies the biological severity of classification errors based on the hierarchy of cell types.
  • Roughness Index (ROGI): Evaluates the "smoothness" of the representation, which indicates how effectively the model captures biological properties for downstream learning.

The benchmark utilizes a diverse range of datasets, from standard pancreas and immune benchmarks to large-scale atlases like the Human Lung Cell Atlas (HLCA) and Tabula Sapiens. To prevent data leakage, the authors also tested the models on AIDA v2, a dataset released after the models' training periods.

The benchmark framework including diverse datasets, evaluated models, and novel ontology-informed metrics.

Key Insights and Results

The evaluation revealed that no single model dominates across all tasks; performance is highly context-dependent.

  • Relational Wisdom: scFMs excel at capturing the relational structure of biology. Even when simple models achieve similar classification accuracy, scFMs' internal maps of cell-type relationships are more biologically accurate.
  • Clinical Utility: In tasks such as cancer cell identification and drug sensitivity prediction, models like scGPT and scFoundation showed strong performance and cross-tissue generalization. scFoundation notably achieved an AUROC of ~0.75 in drug sensitivity tasks.
  • Out-of-Distribution (OOD) Detection: UCE demonstrated a superior ability to identify novel cell types not present in the training data, a vital feature for discovering new biological entities.
  • Representational Smoothness: scFMs generally provide a "smoother" landscape (lower ROGI), making them excellent "plug-and-play" modules for complex downstream tasks.
  • Ensembling for Rare Cells: Combining models (e.g., Geneformer + scGPT) significantly improved accuracy for rare cell types, suggesting that different models capture complementary aspects of biological truth.

Discussion and Future Directions

While scFMs show significant promise, several challenges remain. The high computational cost of these models limits their accessibility. Furthermore, current models are largely restricted to RNA data and predefined gene sets. Future development should focus on integrating multi-omics data (e.g., protein levels, chromatin accessibility) and incorporating more biologically informed pretraining tasks, such as the ontology-guided approach used by scCello. Despite the power of scFMs, traditional methods like scVI or Logistic Regression remain competitive in resource-constrained settings, highlighting the need for a balanced approach in single-cell analysis.

Summary updated

Abstract

This benchmark introduces novel evaluation perspectives, identifying the strengths and limitations of current scFMs, and paves the way for their effective application in biological and clinical research, including cell atlas construction, tumor microenvironment studies, and treatment decision-making.
BibTeX
@inproceedings{wu2025biology,
  author = {Jialu Wu and Qing Ye and Yilin Wang and Renling Hu and Yiheng Zhu and Mingze Yin and Tianyue Wang and Jike Wang and Chang-Yu Hsieh and Tingjun Hou},
  title = {{Biology-driven insights into the power of single-cell foundation models}},
  booktitle = {Genome},
  year = {2025},
  doi = {10.1186/s13059-025-03781-6},
}
Figure