When a computational model screens a thousand drug candidates and a wet-lab team tests the top ten, the usual story is that most of them fail. The file assembled here points to a specific, testable reason hiding inside that failure: the in silico scores are almost always averages over heterogeneous cell populations, while the wet-lab failures are almost always caused by the small minority of cells the average never saw. Bulk measurements average genetic material across millions of cells, hiding rare stem-like and drug-tolerant states that later drive relapse (Genome 2011; Palmer et al.; Lin et al.). Meanwhile, in silico drug prioritization pipelines — even the sophisticated ones that end in real experimental validation — score compounds by how far they shift the average cell state toward health (UNAGI). A compound that converts 95% of cells to a healthy profile looks perfect in silico and then fails in the lab, because the surviving 5% repopulate the tissue.
This proposal makes that mismatch concrete and fixable. I will build a generative perturbation model that predicts the full distribution of post-treatment cell states — not just the mean — for each subpopulation in a heterogeneous sample, and then use it to rank drugs by a tail-eradication score: how much of each subpopulation survives treatment. The central hypothesis is that ranking drugs by the worst-surviving subpopulation, rather than the average response, predicts actual single-cell outcomes — and would have flagged the candidates that mean-shift scoring wrongly promotes. The closest existing work either predicts distributions but never uses them for drug prioritization (scDFM, STATE), or prioritizes drugs but only through mean latent shifts (UNAGI, scPPDM). Nobody has yet connected the two, and the connection is where the validation gap lives.
Core Claim
The claim has two parts, one methodological and one causal. Methodologically: a perturbation model that predicts entire post-treatment cell-state distributions per subpopulation, combined with a worst-case scoring rule, produces drug rankings that are demonstrably different from — and more predictive than — mean-shift rankings on heterogeneous samples. Causally: a measurable fraction of the in silico-to-wet-lab validation gap is attributable to this averaging, and it compounds when the input itself is a bulk profile in which the subpopulations were never resolved in the first place.
The scoring rule is deliberately simple. Let $\mathcal{D}$ be the region of latent space occupied by pre-treatment (diseased) cells, and let $p_i(d)$ be the model's predicted post-treatment distribution of cells in subpopulation $i$ under drug $d$. Define the residual disease mass of subpopulation $i$ as
$$R_i(d) = \int_{\mathcal{D}} p_i(d)(x)\, dx,$$
the fraction of that subpopulation predicted to remain in the diseased region after treatment. Mean-shift scoring effectively minimizes the average $\bar{R}(d)$ across cells. I propose to minimize the worst case,
$$R_\infty(d) = \max_i R_i(d),$$
so a drug is only ranked highly if the model predicts no subpopulation survives in the diseased region. The form comes straight from robust optimization under unknown mixture composition — the honest assumption when the sample's true subpopulation structure is uncertain.
Research Program
Three components, in order of dependency. First, the generative model. I will adapt the conditional flow-matching design of scDFM — a model that learns a velocity field carrying a control cell population to a perturbed one, with a distribution-alignment loss that keeps predicted clouds multi-modal rather than collapsed to their means — and extend its conditioning to take a molecular fingerprint of the drug, following the chemical-encoder pattern of chemCPA (CPA) and scPPDM (scPPDM). scDFM supplies the mechanism I reuse; its own authors show that dropping the distributional loss causes the sharpest accuracy decline, which is the strongest available evidence that distributional detail carries real signal.
Second, the evaluation harness. Tahoe-100M is the ideal testbed because its "Cell Villages" are 3D co-cultures of multiple cancer cell lines treated together with over 1,100 compounds — heterogeneity is built into every sample, and the identity of each cell's line is recoverable. That lets me define ground truth per subpopulation: which cell lines actually persist at which doses. I will measure how often a pipeline declares a drug successful at the village level while ground truth shows at least one constituent line surviving — precisely the masked-minority failure mode that Lin et al. implicate in tamoxifen resistance.
Third, the bulk-masking stress test. For each village I will aggregate the pre-treatment single-cell profiles into a pseudo-bulk mixture and rerun both the mean-shift pipeline and mine with subpopulation proportions recovered only by deconvolution, in the spirit of Count Bridges (which itself shows that discrete-count-aware deconvolution beats STDeconvolve, Jensen-Shannon divergence 0.231 vs. 0.288 on simulated spots). This quantifies how much predictive signal heterogeneity masking destroys at the input stage, separately from how much the mean-shift scoring rule destroys at the output stage. The two error sources are currently conflated in every discussion of the validation gap; separating them is itself a contribution.
Proposed Experiments
The primary experiment trains on a split of Tahoe-100M (95.6 million quality-controlled cells across 17,813 conditions, biological-replicate Pearson correlation 0.975, which sets the noise ceiling) with drugs and doses held out, and asks two questions. How well does the model predict each cell line's post-treatment distribution (per-subpopulation Spearman correlation on differentially expressed genes, plus energy distance between predicted and held-out distributions)? And does $R_\infty$ ranking predict observed subpopulation survival better than mean-shift ranking (area under the receiver operating characteristic curve for "some line survives," computed per drug-dose pair)? The decisive statistic is the disagreement rate between the two rankings among the top-ten drugs: if tail-aware and mean-shift rankings agree, the idea is dead; the file's own logic predicts they will disagree exactly where bulk masking is worst.
A secondary experiment checks generality on genetic perturbations using the Norman 2019 Perturb-seq dataset, where scDFM reports a 19.6% mean-squared-error reduction over the CellFlow baseline, and on drug combinations using ComboSciPlex, where CPA reports $R^2 = 0.81$ for a Panobinostat–Alvespimycin combination and 0.62 for chemCPA on completely unseen drugs. These confirm the distributional backbone remains competitive where tail scoring is not the bottleneck.
A small prospective validation closes the loop. Following the UNAGI template — which predicted nifedipine as anti-fibrotic in pulmonary fibrosis and confirmed it in human precision-cut lung slices, though no reported result found on Lacuna for a quantitative outcome statistic from that validation — I will pick the drug-dose pair where mean-shift and $R_\infty$ rankings disagree most strongly, and test it in a mixed co-culture assay with single-cell readout, scoring whether the minority subpopulation the model flagged actually persists. One decisive wet-lab experiment, chosen adversarially against my own hypothesis, is worth more than ten confirmatory ones.
Baselines
The naive reference point is connectivity-style mean reversal: score each drug by the correlation between the bulk differentially expressed gene signature it induces and the disease signature, effectively what UNAGI does when it steers pathological clusters toward healthy centroids via Connectivity Map profiles. This is the "do nothing sophisticated" comparator because it is what most in silico pipelines actually do, and it inherits every failure mode of static database lookup that Luo et al. document for docking-style approaches. It will be run on identical pseudo-bulk inputs with no tuning budget.
The strongest existing competitor is scPPDM, a latent diffusion model built on a pre-trained single-cell variational autoencoder that conditions on cell state and Morgan chemical fingerprints with factorized guidance. On the Tahoe-100M unseen-drug setting it reports relative gains of +36.11% in Spearman correlation and +34.21% in Pearson correlation for differentially expressed genes, plus a +28.57% lead in top-1000 DEG accuracy — the best numbers in the file for unseen-drug response prediction. I will rerun scPPDM on the same Tahoe training split, same held-out drugs and doses, and same compute, with its own tuning budget matched to mine. Two comparisons follow: head-to-head on per-subpopulation DEG Spearman (its home turf), and head-to-head on subpopulation-survival prediction AUROC, where its standard aggregate metrics have never been evaluated. Beating it means matching it on DEG correlation while significantly exceeding it on survival AUROC and on calibration of the predicted residual mass $R_i$ against observed persistence.
The key ablation runs my own model twice, identical backbone and training, with only the scoring rule changed: mean-shift ranking versus $R_\infty$. This isolates the proposal's central ingredient — the claim is about the scoring rule, not model capacity — and if the ablation shows no difference, the honest conclusion is that the field's problem is resolution, not scoring, and I will say so. A second ablation replaces the flow-matching distributional head with a Gaussian mean-plus-variance head to test whether first two moments suffice.
Why not just…?
Why not just use scPPDM or STATE with their standard metrics? This is the real threat, and it is exactly the ablation above. scPPDM and STATE already predict unseen-drug responses well in aggregate. The counter-evidence is scDFM's demonstration that mean-focused scoring creates artificial performance while missing multi-modal response clouds, and its ablation showing the distributional loss matters most. If running scPPDM with my $R_\infty$ score on top matches the full proposal, the contribution reduces to a metric change on an existing model — still a useful finding, and the experiments are designed so that this outcome is reportable rather than hidden.
Why not just deconvolve the bulk sample first and feed it to existing perturbation models? Deconvolution gives static cell-type fractions, not the ability to simulate anything. POPPY shows the observation side is solved — it maps bulk patient transcriptomes into single-cell foundation-model space with ontology-aware contrastive learning and attributes immunotherapy response to specific cell types — but it is strictly discriminative. The missing piece is forward generative simulation plus worst-case scoring, and the bulk-masking stress test includes the deconvolve-then-perturb pipeline as a baseline so the decomposition is measured, not asserted.
Why not just predict the mean response plus a variance term? Two moments do not capture a drug that splits a population into a large eradicated cloud and a small persistent one — the persistent mode can carry arbitrarily little variance and still be the entire clinical problem. scDFM found that models lacking distributional alignment produced collapsed or artificially shifted distributions precisely in this way, and the Gaussian-head ablation tests the cheap version directly.
Why not just train better bulk models on CCLE and skip single-cell resolution entirely? Classical regression on bulk expression predicts aggregate IC50 values and cannot see the rare resistant subpopulations that drive relapse (Palmer et al.). Empirically, aggregate approaches already lose to heterogeneity-aware ones: TransCell beat LASSO, Elastic Net, Random Forest, and standard deep networks on drug-sensitivity prediction by more than 50%, and SubNetDL significantly outperformed tumor mutational burden in two MSK lung-cancer immunotherapy cohorts ($P = 0.02$ and $P = 6.0\times10^{-3}$). Bulk-only improvements have already hit the ceiling this proposal targets.
Decisive Checks
The idea fails if any of the following hold: per-subpopulation DEG prediction on held-out Tahoe drugs is not competitive with scPPDM's reported unseen-drug performance; the mean-versus-$R_\infty$ ablation shows a disagreement rate below roughly ten percent among top-ranked drugs; or predicted residual mass $R_i$ does not calibrate against observed cell-line persistence across doses. Each check is falsifiable on data already in hand, and the single wet-lab co-culture assay exists to catch whatever all the retrospective checks cannot.
References
- As we come to the end of 2011, several members of the Genome Biology Editorial Board give their views on the state of play in genomics
- A gene expression profile of stem cell pluripotentiality and differentiation is conserved across diverse solid and hematopoietic cancers
- Integrative analyses of gene expression and DNA methylation profiles in breast cancer cell line models of tamoxifen-resistance indicate a potential role of cells with stem-like properties
- A deep generative model for deciphering cellular dynamics and in silico drug discovery in complex diseases
- scDFM: Distributional Flow Matching Model for Robust Single-Cell Perturbation Prediction
- Predicting cellular responses to perturbation across diverse contexts with State
- scPPDM: A Diffusion Model for Single-Cell Drug–Response Prediction
- Predicting cellular responses to complex perturbations in high‐throughput screens
- Tahoe-100M : A Giga-Scale Single-Cell Perturbation Atlas for Context-Dependent Gene Function and Cellular Modeling
- Count Bridges enable Modeling and Deconvolving Transcriptomic Data
- Discovery and identification of potential anti-melanogenic active constituents of Bletilla striata by zebrafish model and molecular docking
- Cross-modal transfer learning for mapping bulk transcriptomes at cellular level
- TransCell: In Silico Characterization of Genomic Landscape and Cellular Responses by Deep Transfer Learning
- A network-based deep learning model integrating subclonal architecture for therapy response prediction in cancer