Defining and benchmarking open problems in single-cell analysis

Malte D. Luecken, Scott Gigante, Daniel B. Burkhardt, Robrecht Cannoodt, Daniel Strobl, Nikolay S. Markov, Luke Zappia, Giovanni Palla, Wesley Lewis, Daniel Dimitrov, Michael Vinyard, D S Magruder, Michaela F. Mueller, Alma Andersson, Emma Dann, Qian Qin, Dominik J. Otto, Michal Klein, Olga Botvinnik, Louise Deconinck, Kai Waldrant, Sai Nirmayi Yasa, Artur Szałata, Andrew Benz, Zhijian Li, Jonathan M. Bloom, Angela Oliveira Pisco, Julio Sáez-Rodríguez, Drausin Wulsin, Luca Pinello, Yvan Saeys, Fabian J. Theis, Smita Krishnaswamy, Swann Floc’hlay, Adam Gayoso, Stephanie Hicks, Yuge Ji, Vitalii Kleshchevnikov, Gioele La Manno, Maximilian G. Lombardo, Romain Lopez, Dario Righelli, Hirak Sarkar, Valentine Svensson, Alexander Tong, Galen Xing, Chenling Xu, Hongyu ZhaoNature Biotechnology 2025 · DOI · OpenAlex

Introduction

The field of single-cell genomics has seen an explosion of computational tools, with over 1,700 algorithms published as of early 2024. However, this rapid growth has led to a "Wild West" of software where benchmarking is often "bespoke"—conducted by the same researchers who developed the tool. This often leads to optimistic evaluations and cherry-picked metrics. Furthermore, traditional static benchmarks become obsolete quickly as new methods and larger datasets emerge. To address this, the Open Problems in Single-Cell Analysis project introduces a living, community-driven platform for the continuous evaluation of single-cell algorithms.

A Living Infrastructure for Benchmarking

The core of the project is a "living" infrastructure designed for automation and reproducibility. Unlike static papers, this platform updates continuously as new methods are submitted, creating a dynamic ecosystem for the field.

The Open Problems ecosystem: a continuous cycle of community contributions, code repository updates, and benchmark results.

The technical framework relies on several key components to ensure neutrality and extensibility:

  • Containerization and Orchestration: Methods are wrapped in Viash components and versioned Docker containers, ensuring cross-platform consistency. Nextflow workflows manage execution on cloud infrastructure (AWS Batch).
  • Continuous Integration (CI/CD): New methods submitted via GitHub are automatically tested. Successful runs update the public leaderboard on openproblems.bio.
  • Standardized Metrics: Performance scores are normalized from 0 (random) to 1 (perfect) to allow for fair comparison across different tasks and datasets.

Scope and Methodology

The platform currently evaluates 171 methods across 12 distinct tasks using 81 diverse datasets spanning multiple species and tissues. These tasks include batch integration, cell-cell communication, and label projection.

Overview of the benchmarking framework, including task visualization (A), the three-step workflow (B), and a sample leaderboard (C).

As shown in the workflow above, the process follows a rigorous three-step cycle:

  1. Prepare Datasets: Curating high-quality data with "proxy" ground truths, such as using spatial co-localization data to validate cell-cell communication inferences.
  2. Run Methods: Executing containerized algorithms in a standardized environment to eliminate the "it works on my machine" problem.
  3. Calculate Metrics: Evaluating performance using community-vetted metrics to ensure unbiased results.

Key Insights and Results

The continuous benchmarking approach has yielded several critical insights that challenge common assumptions in the field:

  • Simplicity vs. Complexity: In tasks like label projection, simple models such as logistic regression often outperformed complex deep learning architectures, particularly when training data contained noise.
  • Cell-Cell Communication (CCC): Evaluation showed that methods focusing on expression magnitude (e.g., CellPhoneDB) generally performed better than those relying solely on specificity.
  • Batch Integration: Deep learning methods like scANVI and scVI were identified as top performers for integrating complex datasets while maintaining biological signal.
  • Brittleness: The "living" nature of the benchmark revealed that some methods are highly sensitive to specific datasets, a nuance often missed in static, one-off evaluations.

Future Directions

While the platform provides a robust foundation, several challenges remain. Future efforts will focus on scaling the infrastructure to handle "atlas-scale" datasets (millions of cells), developing metrics that better capture biological utility rather than just technical accuracy, and expanding the scope to include spatial transcriptomics and perturbation screens. By making these benchmarks citable and community-driven, the project aims to incentivize developers to participate in this evolving ecosystem.

Summary updated

BibTeX
@inproceedings{luecken2025defining,
  author = {Malte D. Luecken and Scott Gigante and Daniel B. Burkhardt and Robrecht Cannoodt and Daniel Strobl and Nikolay S. Markov and Luke Zappia and Giovanni Palla and Wesley Lewis and Daniel Dimitrov and Michael Vinyard and D S Magruder and Michaela F. Mueller and Alma Andersson and Emma Dann and Qian Qin and Dominik J. Otto and Michal Klein and Olga Botvinnik and Louise Deconinck and Kai Waldrant and Sai Nirmayi Yasa and Artur Szałata and Andrew Benz and Zhijian Li and Jonathan M. Bloom and Angela Oliveira Pisco and Julio Sáez-Rodríguez and Drausin Wulsin and Luca Pinello and Yvan Saeys and Fabian J. Theis and Smita Krishnaswamy and Swann Floc’hlay and Adam Gayoso and Stephanie Hicks and Yuge Ji and Vitalii Kleshchevnikov and Gioele La Manno and Maximilian G. Lombardo and Romain Lopez and Dario Righelli and Hirak Sarkar and Valentine Svensson and Alexander Tong and Galen Xing and Chenling Xu and Jonathan M. Bloom and Angela Oliveira Pisco and Julio Saez-Rodriguez and Drausin Wulsin and Luca Pinello and Yvan Saeys and Fabian J. Theis and Smita Krishnaswamy and Hongyu Zhao},
  title = {{Defining and benchmarking open problems in single-cell analysis}},
  booktitle = {Nature Biotechnology},
  year = {2025},
  doi = {10.1038/s41587-025-02694-w},
}
Figure