Machine learning models can now predict how a cell's gene expression will change after a drug or a gene edit, and these predictors are starting to guide real therapeutic programs — including combinations of epigenetic drugs and immune checkpoint blockade that aim to reactivate silenced immune-response genes inside "cold" tumors. The best chemical perturbation model, chemCPA, reaches a median $r^2$ of 0.82 on held-out drugs largely by pre-training on ~1.3 million public bulk expression profiles from the L1000 database Predicting Cellular Responses to Novel Drug Perturbations at a Single-Cell Resolution. But that database is public, and so are the benchmarks the field evaluates on. In language models, exactly this situation — test material statistically present in the training corpus — has been shown to inflate scores by memorization rather than reasoning: models that look state-of-the-art on static benchmarks collapse to near zero on freshly generated test instances DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks, and near-perfect scores on classic benchmarks plunge to 56–70% on post-cutoff material Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?. Nobody has checked whether the same is true for cellular perturbation predictors.
This proposal treats benchmark contamination as the central methodological lens, epigenetic reactivation as the application regime (predicting how HDAC and DNMT inhibitors reawaken silenced immune genes such as Stat1 and Gsdme Pharmacological activation of STAT1-GSDME pyroptotic circuitry reinforces epigenetic immunotherapy for hepatocellular carcinoma), and diminishing returns in data scaling as both a diagnostic and a remedy: the same evidence that says more data stops helping also says more data enlarges the memorization surface.
Core Claim
The headline numbers of cellular perturbation prediction — chemCPA's $r^2 = 0.82$ on sci-Plex3, GEARS' 53% gain on unseen two-gene combinations Predicting transcriptional outcomes of novel multigene perturbations with GEARS — are currently uninterpretable with respect to generalization, for three compounding reasons: the test benchmarks may be partially present in pre-training corpora; train/test splits never hold out an entire mechanism of action (one can always interpolate across similar compounds); and scoring is profile-matching over all genes, which rewards predicting the average post-drug state rather than getting the causal chain right. We claim that under a dynamic, mechanism-held-out evaluation protocol, these scores will drop substantially, and that a compact, influence-selected pre-training corpus recovers much of the lost performance at a fraction of the data — connecting the contamination problem to the scaling plateau rather than treating them as unrelated failures.
The scaling evidence motivating the remedy is already strong. In PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction, scBERT pre-trained on 5 million cells matched or beat scGPT pre-trained on 33 million cells, and zero-shot foundation models were beaten by a simple MLP trained on raw expression — while the graph-based GEARS led by an order of magnitude (AUSPC of roughly 0.008 versus roughly 0.045 on Norman et al., with scGPT barely outperforming the mean predictor). On the language side, Capturing the Temporal Dependence of Training Data Influence shows training data influence collapses into a saturation basin mid-training, so that selecting data only in the high-influence early and late regimes captured 96% of full-selection gains at 5× lower compute; Neuron-Aware Data Selection in Instruction Tuning for Large Language Models shows peak performance at 10–30% of the data, with 10% of Alpaca-GPT4 selected by their method beating the full dataset by 3.24%. Neither selection method has ever been applied to biological data; we propose to port both to the perturbation setting.
Research Program
The first component is a dynamic evaluation generator for chemical and epigenetic perturbations, built on the DyVal template of generating test instances procedurally rather than drawing them from a fixed set DyVal. The combinatorial space is (mechanism class × compound × dose × cell line × perturbation combination). The critical design choice is mechanism-level holdout: entire pharmacological classes — all class-I HDAC inhibitors, or all DNMT inhibitors — are removed from training, so test-time prediction of, say, CXD101-like activity cannot rely on a sibling compound seen during training. Difficulty is then scaled in DyVal's spirit (its D1–D4 tiers, where GPT-3.5-Turbo lost 23% accuracy) by increasing dose combinations and predicting two-compound synergy, which is where prior models degrade most.
The second component is a contamination audit. Following the membership-inference logic used to show that historical causal benchmarks were leaked Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?, we adapt Min-K%-style membership scores from token likelihoods to expression-profile likelihoods, and measure how strongly Norman et al. (2019), Replogle et al. (2022), and sci-Plex3 test profiles are flagged as members of the pre-training corpora of scGPT (33M cells), scBERT (5M), Geneformer, and chemCPA's L1000 pre-training set. We then compare model performance on test perturbations flagged as leaked versus clean — the biological analogue of the COPA-versus-CausalProbe plunge.
The third component replaces the scoring function. Rather than $r^2$ over all genes, we score direction-of-effect along the causal cascade that the epigenetic-immunotherapy literature has established: HDAC inhibition → reactivation of IFNγ-responsive genes → Gsdme upregulation → pro-inflammatory pyroptosis markers → immune-signature shifts in co-culture composition. This template comes from the STAT1-GSDME circuitry paper, where the combination of the class-I HDAC inhibitor CXD101 and anti-PD-L1 achieved a >50% cure rate in resistant mouse models and the effect was abolished by knocking out Stat1 or Gsdme Pharmacological activation of STAT1-GSDME pyroptotic circuitry..., and from the MYBL2–CCL2 axis in ovarian cancer, where a defined transcriptional program recruits immunosuppressive macrophages and CDK2 blockade breaks it The MYBL2–CCL2 axis.... A model that predicts the average post-drug vector with high $r^2$ but gets the direction of Stat1/Gsdme/interferon-response genes wrong fails this test; that failure mode is invisible to current metrics.
The fourth component is the remedy: influence-selected pre-training. We port the trajectory-specific leave-one-out influence of DVEmb Capturing the Temporal Dependence of Training Data Influence and the activation-geometry selection of NAIT Neuron-Aware Data Selection... to chemCPA's pre-training pipeline, selecting 10% and 30% subsets of L1000 by influence and by activation-shift PCA, and measure whether the scaling plateau and the contamination exposure shrink together.
Proposed Experiments
Experiment 1 evaluates chemCPA on sci-Plex3 (~650,000 cells, 188 compounds, three cancer lines) under mechanism-held-out splits, holding out HDAC inhibitors and DNMT inhibitors entirely. Its published reference points are a median $r^2$ of 0.82 over all genes and 0.79 over differentially expressed genes at 10 µM, against 0.48 and 0.64 for the non-pretrained baseline Predicting Cellular Responses to Novel Drug Perturbations.... We report both the standard random-split number (reproduced under our compute budget) and the mechanism-held-out number, plus a difficulty curve over dose-combination tiers. Experiment 2 runs the membership-inference audit across the foundation-model corpora and compares predictions on flagged-clean versus flagged-leaked test perturbations from Norman et al. (131 two-gene combinations) and Replogle et al. (>2.5 million cells). For Replogle, no reported result found on Lacuna for a numerical baseline beyond the GEARS/PertEval evaluations; our audit supplies the first clean-versus-leaked comparison. Experiment 3 retrains chemCPA with 10% and 30% influence-selected L1000 subsets and evaluates on the dynamic suite, measuring performance, pre-training cost, and membership-inference exposure on held-out mechanisms simultaneously. For all generated test conditions we follow the statistical design guidelines of scPerturb: Harmonized Single-Cell Perturbation Data — at least 300 cells per perturbation and ~1,000 UMI counts per cell — so that evaluation signal separates from sampling noise. Experiment 4, deliberately scoped as a stretch goal, scores cascade-direction on immune-signature genes after epigenetic drugs in the co-culture direction; we acknowledge that scPerturb's 44 datasets are overwhelmingly monocultures, so full TME prediction is out of reach today and we limit ourselves to direction-of-effect on the macrophage- and T-cell-signature genes defined by the MYBL2–CCL2 and STAT1–GSDME studies.
Baselines
The do-nothing reference is the mean baseline, the predictor that outputs the average post-perturbation cell state for every drug; on Norman et al., scGPT (AUSPC ~0.045) barely outperformed it while GEARS (~0.008) led by an order of magnitude PertEval-scFM, which makes the mean predictor the exact failure mode our cascade scoring must expose. It will be run on identical splits and doses; any serious model must beat it on perturbation-specific signal before we discuss anything else. The strongest existing competitor is chemCPA as published: pre-trained on the full L1000 corpus and fine-tuned on sci-Plex3, reporting median $r^2 = 0.82$ on all genes and 0.79 on DEGs at 10 µM Predicting Cellular Responses to Novel Drug Perturbations.... We will run it under the same data, the same fine-tuning compute, and the same tuning budget as every variant we train, so that any difference is attributable to the evaluation protocol or the pre-training corpus, not to budget. Beating it means one of two outcomes: matching its random-split $r^2$ while significantly exceeding it on mechanism-held-out dynamic splits, or matching its dynamic-split performance using at most 30% of the pre-training corpus. The third baseline is GEARS, the strongest combinatorial extrapolator on the genetic side, with a 53% improvement on zero-shot two-gene combinations and 40% higher precision on genetic-interaction subtypes GEARS; we run it on Norman and Replogle under our splits, while noting honestly that it handles genetic perturbations, not compounds, so it competes only on the genetic portions of the audit. Finally, the key ablation isolates the proposal's central ingredient: chemCPA with an identical architecture and training budget, but with a decontaminated and influence-selected L1000 pre-training corpus. If this ablation matches full-corpus chemCPA on dynamic splits while shrinking membership-inference exposure, the data-selection mechanism — not architecture — is doing the work.
Why not just…?
Why not just use the SPECTRA splits that already exist? SPECTRA, from PertEval-scFM, varies split difficulty by sparsifying the gene-gene graph within genetic perturbation data in cell lines, and we reuse it for split difficulty. But it was built for CRISPR perturbations where genes are "knocked out" by zeroing input values; it never measures whether test profiles leaked into pre-training corpora, never holds out a chemical mechanism class, and never asks whether the predicted response preserves a causal cascade. Experiments 1 and 2 directly test the two things SPECTRA cannot, and if chemCPA's $r^2$ is unchanged under mechanism-held-out splits and no leakage is detected, our central claim is falsified — a clean, publishable negative result that the field's numbers are trustworthy.
Why not just collect more cells? This is the most serious alternative, and the evidence against it is direct: scBERT pre-trained on 5 million cells matched or beat scGPT on 33 million PertEval-scFM; DVEmb shows data influence collapsing into a mid-training saturation basin Capturing the Temporal Dependence...; and NAIT shows peak performance at 10–30% of instruction data Neuron-Aware Data Selection.... Scaling also enlarges the contamination surface, since more public profiles means a higher chance that any given benchmark is in the corpus. Experiment 3 settles this empirically: if the full corpus beats the selected subsets on dynamic splits, the scaling optimists win and we will report it.
Why not just use GPT-4 as the data selector or judge? External LLM scoring was measured at 19× the cost of the internal-activation method NAIT Neuron-Aware Data Selection..., and frontier models are unreliable judges of causal structure — Claude 3 Opus scored under 70% on deceptive causal pairs and 33.8% of GPT-4's structured-reasoning failures were unsubstantiated responses Unveiling Causal Reasoning...; DyVal. More fundamentally, an LLM cannot score a 978-gene expression vector against a held-out mechanism; the quantities we need judged are numerical, and influence functions compute them exactly.
Why not just do the wet-lab experiment — maximum-dose pan-HDAC inhibitor plus anti-PD-1? Because the biology says the mechanism is tumor-specific: CXD101 plus anti-PD-L1 cured over 50% of resistant mouse hepatocellular carcinomas, but with uncharacterized off-target effects and fibrotic barriers the authors themselves flag Pharmacological activation of STAT1-GSDME...; in ovarian cancer resistance runs through a CDK2–MYBL2–CCL2 macrophage-recruitment axis that HDAC inhibition does not touch The MYBL2–CCL2 axis...; and hyaluronic-acid stroma physically excludes T cells in gastric cancer Human Hyaluronidase PH20.... Screening every mechanism combination in vivo is infeasible, which is exactly why a trustworthy in silico ranking layer matters — and why its contamination audit must come first.
Decisive Checks
The proposal stands or falls on three falsifiable outcomes. First, if chemCPA's $r^2$ on sci-Plex3 survives mechanism-held-out splits essentially unchanged, and membership inference flags negligible leakage, then the perturbation field is cleaner than language modeling and this work becomes the audit that certified it. Second, if models with high profile-level $r^2$ systematically fail the cascade-direction score, current metrics are rewarding average-state prediction and the field's model rankings need revision. Third, if influence-selected 10–30% pre-training corpora match full-corpus performance on dynamic splits while reducing measured leakage, then the scaling plateau and the contamination problem are two faces of the same redundancy — and the practical prescription for building the next generation of perturbation predictors is curate, don't accumulate.
References
- Predicting Cellular Responses to Novel Drug Perturbations at a Single-Cell Resolution
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Pharmacological activation of STAT1-GSDME pyroptotic circuitry reinforces epigenetic immunotherapy for hepatocellular carcinoma
- Predicting transcriptional outcomes of novel multigene perturbations with GEARS
- PertEval-scFM: Benchmarking Single-Cell Foundation Models for Perturbation Effect Prediction
- Capturing the Temporal Dependence of Training Data Influence
- Neuron-Aware Data Selection in Instruction Tuning for Large Language Models
- The MYBL2–CCL2 axis promotes tumor progression and resistance to anti-PD-1 therapy in ovarian cancer by inducing immunosuppressive macrophages
- scPerturb: Harmonized Single-Cell Perturbation Data
- Human Hyaluronidase PH20 Potentiates the Antitumor Activities of Mesothelin-Specific CAR-T Cells Against Gastric Cancer