TL;DR
This paper introduces STACK, a novel foundation model for single-cell biology that leverages the context of neighboring cells to generate more accurate and informative representations. STACK enables "in-context learning," allowing it to predict cellular responses to unseen conditions without retraining, and has been used to create Perturb Sapiens, a comprehensive atlas of perturbed human cells.
What Problem Does This Paper Address?
Single-cell RNA sequencing (scRNA-seq) generates vast amounts of data, promising deep insights into cellular diversity. However, existing computational models often treat cells in isolation, overlooking crucial contextual information that can improve measurement precision. This limitation hinders their ability to generalize to new biological conditions or tasks without extensive retraining. Specifically, current foundation models struggle with "zero-shot" learning (performing well on unseen data) and accurately predicting the effects of perturbations or variations in cellular states.
Methods
STACK employs a tabular transformer architecture that processes cells within their context. Key innovations include:
- Gene Module Tokens: Instead of processing individual genes, STACK tokenizes gene expression into "gene module tokens," representing coherent subsets of gene variability.
- Tabular Attention: This architecture uses multi-head attention (MHA) layers that operate both intra-cellularly (within a cell's tokens) and inter-cellularly (across tokens of cells within the same context).
- Masked Gene Reconstruction: During pre-training, STACK learns by reconstructing masked gene expression values, leveraging the context of other cells in the set.
- In-Context Learning (ICL): After pre-training, STACK is fine-tuned to perform "cell prompting." This allows it to predict how query cells would behave under a condition specified by "prompt" cells, enabling zero-shot prediction of perturbation effects or donor-specific states. This is achieved through a self-distillation procedure and a novel post-training objective.
Datasets
- Pre-training: STACK was pre-trained on scBaseCount, a massive dataset of 149 million human single cells.
- Post-training (ICL): For in-context learning, STACK was post-trained on a curated dataset of 55 million cells from CELLxGENE and the Parse PBMC 10M dataset, which includes cytokine perturbations.
- Evaluation: The model was evaluated on numerous observational and perturbational datasets, including Kidney atlas, Brain atlas, Lymph node atlas, Lung atlas, Tabula Sapiens, OpenProblems (drug perturbations), Tahoe-100M, Parse-PBMC, and X-Atlas (Xaira).
- Perturb Sapiens: This paper introduces Perturb Sapiens, a novel whole-organism atlas of perturbed cells generated using STACK, spanning 28 tissues, 40 cell classes, and 201 perturbations.
Key Insights and Ideas
- Context is King: The paper strongly demonstrates that considering cells within their biological context significantly improves representation learning and downstream task performance, especially in zero-shot scenarios.
- In-Context Learning for Biology: STACK pioneers in-context learning for single-cell biology, enabling models to adapt to new conditions and tasks at inference time without retraining, akin to large language models.
- Perturb Sapiens as a Resource: The creation of Perturb Sapiens highlights STACK's capability to generate comprehensive, high-resolution atlases of cellular responses, providing a valuable resource for biological research.
- Generalization Beyond Supervision: STACK's ability to generalize to unseen cell types, tissues, and perturbations without explicit labels or fine-tuning is a major step towards truly universal biological foundation models.
Key Results
- Superior Zero-Shot Performance: STACK consistently outperforms baseline models (including zero-shot, fine-tuned, and trained-from-scratch models) on various downstream tasks, achieving state-of-the-art results in 28 out of 31 perturbation prediction benchmarks.
- Enhanced Embeddings: STACK embeddings show substantial improvements in cell type classification and integration tasks, outperforming alternatives by significant margins (e.g., +139.4% in integration performance on Tabula Sapiens).
- Accurate Perturbation Prediction: STACK accurately predicts perturbation effects, even in novel cell types and across different datasets, outperforming existing methods by approximately 100% on large-scale datasets.
- Perturb Sapiens Atlas: The generated Perturb Sapiens atlas, covering 28 tissues and 201 perturbations, reveals realistic cellular responses and has been validated against in vitro data.
Limitations and Future Work
- Human-Centric Model: The current STACK model is trained exclusively on human data. Extending it to multi-species applications would require modifications to the tokenization procedure to handle gene misalignment.
- Rare Cell Types and Weak Effects: Model calibration for rare cell types and subtle perturbation effects remains an area for improvement.
- Signaling Cascades: The complex, time-dependent nature of signaling cascades might complicate the interpretation of STACK predictions, especially for secondary perturbation effects.
- Data Curation: While post-training focused on in vivo cell types, more sophisticated data curation and alignment schemes could further enhance generalization across diverse experimental settings.
Summary updated