Assessing Scale and Predictive Diversity in Models for Single-Cell Transcriptomics based on Geneformer

Junfan Chen, Fabian Schmidt, Ricardo HenaobioRxiv 2025 · DOI

TL;DR

This paper challenges the "bigger is better" mantra for single-cell transcriptomics foundation models, finding that scaling up pre-training data doesn't always lead to better downstream performance. Instead, architectural improvements that enhance predictive diversity and biological plausibility are crucial for advancing the field.

What Problem Does This Paper Address?

The field of single-cell transcriptomics is increasingly leveraging "foundation models" – large, pre-trained models that learn generalizable representations from vast datasets. These models hold promise for simplifying complex downstream analyses like cell type classification and understanding cellular responses. However, two key questions have remained:

  1. Does simply increasing the amount of pre-training data (scaling) reliably improve model performance on various single-cell tasks?
  2. Do models trained on "rank-ordered" gene expression profiles (like Geneformer) offer an advantage in dealing with "batch effects" – technical variations that can obscure biological signals?

The authors aim to provide empirical evidence to answer these questions and guide future development in this rapidly evolving area.

Methods

The study systematically evaluates transformer-based models, primarily focusing on the Geneformer (GF) architecture, which represents gene expression as ranked sequences. They investigate two main aspects:

  • Scaling: They compare models pre-trained on datasets of different sizes (1 million vs. 30 million single-cell profiles).
  • Architectural Refinements: They introduce and evaluate a modified architecture called GFCAB (Geneformer with Cumulative-assignment on the prediction Adjustment and Boosting on the similarity-based diversity). This model incorporates two key mechanisms:
  • Cumulative Prediction Adjustment: Aims to reduce redundancy in gene predictions within a single cell profile.
  • Similarity-Based Regularization: Encourages the model to predict a more diverse set of genes, moving beyond a bias towards frequently expressed genes.

The models are evaluated on pre-training objectives (masked gene prediction accuracy, repetition, and uniqueness) and a suite of downstream tasks including cell type classification and zero-shot batch-effect mitigation.

Datasets

The study utilizes several datasets for training and evaluation:

  • Pre-training:
    • Genecorpus-30M: A large dataset of approximately 30 million ranked single-cell transcriptomic profiles.
  • Genecorpus-1M: A smaller dataset created by randomly sampling 1 million profiles from Genecorpus-30M.
  • Downstream Evaluation:
    • Classification Tasks:
      • Gene dosage-sensitivity classification (using data from Theodoris et al. [6]).
  • Tissue-specific cell type classification (using data from Theodoris et al. [6]).
  • Cardiomyocyte disease-state classification (using data from Chaffin et al. [24]).
  • Zero-Shot Batch-Effect Mitigation Tasks: * Immune 330K dataset.
  • Pancreas 16K dataset.
  • PBMC 12K dataset.
  • scCello OOD cell type 487K dataset.

Key Insights and Ideas

  • "Bigger is not always better": Scaling up pre-training data improved masked gene prediction accuracy but didn't consistently translate to better performance on downstream tasks like classification or batch correction.
  • Diminishing returns from scale: Smaller models often matched or even outperformed larger ones, suggesting that the benefits of increased data size plateau or even decrease beyond a certain point.
  • Rank-based models struggle with batch effects: Models trained on rank-ordered profiles, while useful for other tasks, did not inherently offer advantages for mitigating batch effects and often underperformed specialized methods.
  • Architectural innovation is key: The proposed GFCAB architecture, with its focus on cumulative prediction adjustment and similarity-based regularization, demonstrated improved predictive diversity and better performance, especially on smaller datasets, without relying solely on massive data scales.
  • Diversity matters: High redundancy in predicted genes remains a significant limitation in current models, hindering biological interpretability. Enhancing predictive diversity is a critical area for future progress.

Key Results

  • Pre-training: Models trained on smaller datasets (1M profiles) showed significantly higher prediction diversity (uniqueness) and lower repetition compared to models trained on larger datasets (30M profiles), while requiring substantially less computational resources (e.g., ~7.8% of resources for 1M vs. 30M).
  • Classification: For classification tasks, models pre-trained on larger datasets (30M) generally performed better, highlighting the benefit of scale for fine-grained discrimination. However, GFCAB-1M achieved performance comparable to GF-30M in several benchmarks, demonstrating the power of architectural design.
  • Batch-Effect Mitigation: Scaling pre-training data did not improve batch-effect mitigation performance; in fact, smaller models often performed better. GFCAB showed marked improvements in this area compared to GF, narrowing the gap with specialized domain-specific methods like scVI, though still trailing them.
  • Ablation Study: Both cumulative prediction adjustment and similarity-based regularization in GFCAB contributed to improved diversity and performance, with GFCAB (combining both) showing the strongest results, particularly in batch-effect evaluations.

Limitations and Future Work

  • Batch Correction Limitations: Despite architectural improvements, GFCAB and GF still underperformed compared to domain-specific methods like scVI for batch correction. This suggests that rank-based normalization alone is insufficient for robust batch handling, and future models may need representations explicitly designed for cross-batch alignment.
  • Generalizability: The study focused on specific architectures and datasets. Further research is needed to explore whether these findings generalize across other single-cell modalities and tissues.
  • Future Directions: The authors suggest pursuing hybrid strategies that combine rank-based modeling with batch-sensitive objectives, exploring multimodal integration, and investigating the interplay between scale and architecture across different biological contexts. They emphasize that future progress will likely hinge on architectures and objectives that faithfully represent the statistical and biological properties of single-cell data, rather than solely on increasing data scale.

Summary updated

Abstract

Abstract Foundation models are increasingly applied to single-cell transcriptomics, where they promise to capture generalizable representations that support diverse downstream analyzes. However, two central questions remain: Does scaling pre-training data reliably improve performance, and do models trained on rank-ordered expression profiles confer advantages for mitigating batch effects? We addressed these questions by systematically assessing transformer-based models pre-trained on ranked single-cell profiles with varying data scales. The models were evaluated on masked gene prediction and downstream tasks, including cell type classification, perturbation response prediction, and zero-shot batch integration. To complement prediction accuracy, we further quantified prediction repetition, uniqueness, and diversity. In addition, we evaluated architectural refinements that incorporate cumulative prediction adjustment and similarity-based regularization; however, the study mainly focused on comparative evaluations rather than the development of new models. Our results indicated that scaling pretraining corpora improved masked prediction accuracy but did not consistently enhance downstream performance. Smaller models often matched or exceeded larger ones, indicating diminishing returns relative to scale. Rank-based models offered limited robustness to batch effects and consistently underperformed relative to domain-specific correction methods. Across all scales, high redundancy in predicted genes remains a major limitation. Together, these findings challenge assumptions that larger datasets or rank-order modeling automatically confer stronger generalization. Progress in single-cell foundation models may depend less on scale and more on pre-training objectives that enhance predictive diversity and biological plausibility. Author summary In recent years, large machine learning models have been adapted to study single cells, raising hopes that simply training on more data would automatically lead to better biological insights. Our study aimed to test this assumption by carefully evaluating models trained on small versus very large collections of single-cell data. We also asked whether models that learn from ranked lists of gene expressions offer an advantage in correcting technical differences between experiments, a common challenge in biology. We found that bigger is not always better. Although larger models improved performance in predicting missing genes, this advantage did not consistently translate to stronger results in tasks such as cell type classification or data integration across studies. In fact, smaller models often matched or outperformed their larger counterparts. Models trained on ranked data were also not sufficient to reliably correct batch effects, and traditional tools still worked better for this purpose. Our findings suggest that future progress may come less from ever-larger models and more from rethinking how models learn from biological data. This work provides guidance for researchers designing new approaches and helps clarify realistic expectations for applying artificial intelligence to single-cell biology.
BibTeX
@misc{chen2025assessing,
  author = {Junfan Chen and Fabian Schmidt and Ricardo Henao},
  title = {{Assessing Scale and Predictive Diversity in Models for Single-Cell Transcriptomics based on Geneformer}},
  year = {2025},
  doi = {10.1101/2025.11.04.686458},
  howpublished = {bioRxiv},
}
Figure