PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis

Zichao Yan, Yan Wu, Esther Wershof, Sebastian M. Schmon, Marcel Nassar, Blazej Osinski, Ridvan Eksi, Kun Zhang, Thore Graepel

Abstract

PerturBench is a modular benchmarking framework designed to standardize the evaluation of machine learning models predicting cellular responses to chemical and genetic perturbations. By curating diverse large-scale single-cell datasets and introducing rank-based metrics, the authors identify critical failure modes like "mode collapse" in complex architectures. Their findings suggest that simpler models often outperform sophisticated ones as data scales, and that modular ablations can significantly improve model performance.

Introduction

In drug discovery and genetics, the astronomical search space of chemical and genetic perturbations—roughly 20,000 genes and $10^{60}$ drug-like molecules—precludes exhaustive laboratory testing. Generative machine learning models aim to bridge this gap by predicting "counterfactual" cellular states: the response of a cell to a perturbation it has not yet encountered. However, the field has lacked standardized benchmarks, leading to inconsistent comparisons and models that often appear successful while failing to capture specific perturbation effects.

The PerturBench Framework

To address these challenges, the authors introduce PerturBench, a modular codebase built on PyTorch Lightning and Hydra. The framework standardizes the research pipeline into three core modules: data curation, model implementation, and analysis. The "Model Zoo" includes re-implementations of state-of-the-art models like Compositional Perturbation Autoencoders (CPA) and SAMS-VAE, alongside strong baselines. The framework supports diverse modeling approaches, including control matching and disentanglement strategies.

The PerturBench framework overview, illustrating the pipeline from single-cell datasets to standardized benchmarks.

Novel Evaluation Metrics

A key contribution of PerturBench is the introduction of rank-based metrics. Traditional metrics like Root Mean Square Error (RMSE) often fail to detect "mode collapse," a failure state where a model ignores specific inputs and outputs a generic "average" response. Rank-based metrics assess whether a model can correctly order perturbations by their similarity to a reference, providing a more sensitive measure of prediction fidelity and exposing models that learn generic rather than specific responses.

Visualization of rank-based metrics, which evaluate a model's ability to correctly order predicted perturbations relative to a reference.

Key Insights and Results

  • Simplicity at Scale: On the largest datasets, such as Jiang24 (1.6 million cells), simpler architectures like "Latent Additive" or "Decoder-Only" models frequently outperformed complex Variational Autoencoders (VAEs). This suggests that as data volume increases, the rigid inductive biases of complex models may become a hindrance.
  • The Value of Ablation: Modular testing revealed that removing the adversarial classifier from the popular CPA model actually improved its ranking performance on the Srivatsan20 dataset, highlighting the importance of testing individual components.
  • Combinatorial Prediction: For predicting the effects of dual genetic edits (Norman19), simple additive models proved highly competitive, matching or exceeding the performance of more specialized architectures like GEARS.
  • Foundation Models: The use of embeddings from foundation models like scGPT provided only marginal gains, indicating that task-specific architecture and data quality remain the primary drivers of performance.

Conclusion

PerturBench establishes a rigorous standard for cellular perturbation modeling. By exposing common failure modes and demonstrating the effectiveness of simpler architectures at scale, it provides a roadmap for future research. The open-source nature of the framework invites community-driven expansion to incorporate biological priors and tackle the ongoing challenge of out-of-distribution generalization.

Summary updated

BibTeX
@inproceedings{yan2025perturbench,
  author = {Zichao Yan and Yan Wu and Esther Wershof and Sebastian M. Schmon and Marcel Nassar and Blazej Osinski and Ridvan Eksi and Kun Zhang and Thore Graepel},
  title = {{PerturBench: Benchmarking Machine Learning Models for Cellular Perturbation Analysis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=PPPDuyiZaG},
}
Figure