Attribution-based diagnostics for single-cell foundation models

Farnoush Rezaei Jafari, Tristan Lazard, Kenza Bouzid, Shruthi Bannur, Jeremiah Wander, Ashley Mae Conard, Stephanie Hyland, Lorin Crawford, Daniel Coelho de Castro

TL;DR

This paper introduces a post-hoc diagnostic framework using Layer-wise Relevance Propagation (LRP) to "peek under the hood" of single-cell RNA foundation models (scFMs). By tracing predictions back to specific input genes, the authors demonstrate that attribution can distinguish between models that have learned genuine biology and those relying on "shortcuts," providing a much-needed sanity check for in-silico biological discovery.

What Problem Does This Paper Address?

Single-cell foundation models (scFMs) like scGPT and Cell2Sentence are increasingly used to predict cell types and how genes respond to perturbations. However, high predictive accuracy can be deceptive. A model might achieve great results by exploiting "spurious correlations" or technical shortcuts (like the rank of gene expression) rather than capturing the underlying biological mechanisms.

If we want to use these models for drug discovery or understanding disease, we need to know why they make a prediction. The challenge is that these models are complex "black boxes." This paper addresses the gap between raw performance metrics and biological plausibility, providing tools to audit whether a model's internal logic aligns with established biological knowledge, such as marker genes and gene regulatory networks (GRNs).

Methods

The researchers employ a suite of interpretability techniques to audit two major tasks:

  1. Cell-Type Annotation: They used Linear Probes—simple classifiers trained on the frozen representations of models like Cell2Sentence (C2S) and scGPT. To explain these classifications, they used Layer-wise Relevance Propagation (LRP), which decomposes the output prediction into "relevance scores" for every input gene.
  2. Contextual Analysis: They utilized Symbolic XAI to test higher-order interactions. This allows them to ask logical questions like: "Does this gene's importance depend on the presence of other specific genes?"
  3. Perturbation Auditing: For the more complex task of predicting genetic perturbations, they combined scGPT with a STATE (State Transition) architecture. They then compared the model's attributions against a known ground-truth regulatory network in a synthetic setting.
Overview of the attribution framework for cell typing and perturbation prediction.

Datasets

The study utilizes both real-world and synthetic data:

  • CZ CELLxGENE Census: Two curated benchmarks—one for blood (PBMC) with ~61k cells and 22 subtypes, and one for the human brain with ~61k nuclei across 7 broad classes.
  • Synthetic Perturbation Dataset: Because real-world "ground truth" for gene regulation is often incomplete, the authors simulated a system of 128 genes using stochastic differential equations (SDEs). This provided a fully known Gene Regulatory Network (GRN) to act as a "gold standard" for auditing the model.

Key Insights and Ideas

  • Attribution as a Quality Signal: The authors found that when a model makes a correct prediction, its attribution scores align much more closely with known marker genes than when it makes an error. This suggests that "wrong" predictions are often driven by non-biological signals.
  • The "Shortcut" Warning: In the Cell2Sentence model, relevance was heavily concentrated in the first few genes of the input sequence (which are sorted by expression). While these are often informative, it highlights a potential "positional bias" where the model might over-rely on highly expressed genes simply because of their placement in the sequence.
  • Context is King: Using symbolic queries, the researchers showed that shared marker genes (genes that appear in multiple similar cell types) only become "informative" to the model when considered alongside the rest of the cellular context.
Relevance of shared markers changes significantly when considering the full cellular context.
  • Early Learning of Logic: In perturbation tasks, the model began to learn the correct regulatory logic (which genes affect which others) quite early in training—often before that logic was clearly reflected in the final predicted expression values.

Key Results

  • Marker Recovery: LRP-based attribution significantly outperformed simple gene expression values in identifying marker genes across all models and tasks (see Figure 2).
  • Regulatory Alignment: In the synthetic perturbation task, Perturbation-LRP successfully recovered direct regulatory edges (Average Precision ~0.14-0.15) and showed significant alignment with the "True GRN" compared to randomized controls ($p=0.001$).
  • Prediction Reliability: There was a strong correlation ($\rho = 0.897$) between the amount of relevance the model assigned to the actual perturbation input and the overall quality of its prediction. This means attribution can act as a "confidence score" for individual predictions.
Evaluation of how well model attributions recover the true regulatory network.

Limitations and Future Work

  • Ground Truth Scarcity: The study highlights that our biological "reference manuals" (marker panels and GRNs) are still incomplete, which limits how thoroughly we can audit these models on real-world data.
  • Synthetic Simplicity: While the synthetic dataset provided a perfect ground truth, it lacks the full complexity of real biological systems.
  • Future Directions: The authors suggest that attribution shouldn't just be a post-hoc "autopsy" tool. Instead, it could be integrated into the training process itself to penalize models that rely on non-biological shortcuts, leading to more robust and trustworthy foundation models.

Summary updated

Abstract

Single-cell RNA foundation models (scFMs) are increasingly used for in-silico biological discovery, yet predictive performance alone does not reveal whether their decisions rely on biologically meaningful signals or model-specific shortcuts. We systematically investigate post-hoc attribution of downstream predictions across two tasks: cell-type annotation using pretrained Cell2Sentence and scGPT models, and perturbation prediction using scGPT+STATE in a controlled setting. For cell annotation, we use attribution to assess whether predictions rely on established marker genes and how their relevance varies with model architecture, expression rank, and the surrounding cellular context. For perturbation prediction, we compare attribution patterns with known regulatory ground-truth in a synthetic setting to assess what they reveal about the model's learned computations. Across these tasks, attribution reveals biological feature use, contextual interactions, contributions from different input pathways, learning progression, associations with prediction quality, and potential pitfalls that are not fully captured by standard performance metrics. Experiments with complete and limited regulatory knowledge further demonstrate that conclusions drawn from attribution depend strongly on the choice of biological reference, baseline, and null model. Together, these findings highlight the value of attribution as a flexible post-hoc diagnostic for understanding model behavior, identifying potential issues, and informing the development of scFMs.
BibTeX
@misc{jafari2026attribution,
  author = {Farnoush Rezaei Jafari and Tristan Lazard and Kenza Bouzid and Shruthi Bannur and Jeremiah Wander and Ashley Mae Conard and Stephanie Hyland and Lorin Crawford and Daniel Coelho de Castro},
  title = {{Attribution-based diagnostics for single-cell foundation models}},
  year = {2026},
  eprint = {2611.02335},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02335},
}
Figure