Genetic combination screens look for pairs of genes whose joint knockout does something neither knockout does alone. There are about two hundred million possible gene pairs, so no laboratory can test them all. Current active-learning systems therefore train a predictor on the pairs measured so far and send to the lab the pairs with the highest predicted effect. This proposal argues that the step "pick the highest predicted value out of an enormous pool" produces overconfidence by itself, even when the predictor is well calibrated on an average pair. When you take the maximum of millions of noisy predictions, the winners are mostly the pairs whose errors happen to be largest and positive. The predicted synergy of the selected batch is therefore biased upward, and the bias grows with the size of the pool. The search space explosion does more than make search expensive: it turns ordinary prediction noise into systematic overconfidence on exactly the candidates that get tested.
We propose to measure this selection-induced overconfidence directly and to correct it. The correction is a shrinkage of the predicted interaction term whose strength depends on two quantities: how large the candidate pool is, and how much data supports each gene in the pair. We calibrate this shrinkage on "pseudo-rounds" carved out of the already-collected data, so it costs no extra experiments. The claim to test is that a predictor calibrated on the batches it actually selects finds more true hits per round than both pure exploitation and generic uncertainty-aware exploration.
Core Claim
The strongest current method for combinatorial perturbation discovery, NAIAD (Active Learning for Efficient Discovery of Optimal Combinatorial Perturbations), combines two parts. A linear branch models the additive effect of each single gene. A permutation-invariant neural branch models the deviation from additivity, meaning synergy or buffering. NAIAD selects pairs by Maximum Predicted Effect (MPE), which is pure exploitation with no uncertainty estimate.
The calibration literature says high-capacity components become overconfident: BERT is more accurate than shallower models but far worse calibrated (Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data). Miscalibration has also been shown to waste active-learning budget in image classification (CALICO: Confident Active Learning with Integrated Calibration).
What neither line addresses is that calibration measured on random held-out pairs is the wrong target for a search procedure. What matters is calibration conditional on being selected. Under a simple Gaussian model the gap has a known form. If the interaction branch's prediction errors have standard deviation $\sigma$ across $N$ roughly independent candidates, the largest of $N$ standard Gaussians grows like $\sqrt{2\ln N}$ (a standard extreme-value result). The expected optimism of the top-ranked candidate is therefore approximately
$$ \mathbb{E}[\hat{y}_{\text{sel}} - y_{\text{sel}}] \approx \sigma \sqrt{2 \ln N}, $$
where $\hat{y}_{\text{sel}}$ is the predicted and $y_{\text{sel}}$ the realized effect of the selected pair.
The prediction we test is that this optimism lives mainly in the interaction branch, not the additive branch. It should be largest for pairs whose genes have few single-perturbation measurements, since that is where $\sigma$ is large. This is exactly the regime NAIAD highlights, where each gene is seen only about four times. If the prediction holds, the fix is targeted. We replace the raw prediction with
$$ \hat{y}_{ij} = a_i + a_j + \lambda(n_i, n_j, N)\, g_{ij}, $$
where $a_i$ is the additive effect of gene $i$, $g_{ij}$ is the interaction branch's output for the pair, $n_i$ is the number of measurements involving gene $i$, and $\lambda$ is a learned shrinkage factor between 0 and 1. We pair this with an interval whose coverage is checked on selected batches, not on random pairs.
Research Program
The first part is diagnostic. We replay active learning retrospectively on the combinatorial CRISPR datasets NAIAD used: the Norman et al. Perturb-seq pairs, the Horlbeck et al. fitness screen with more than 100,000 double knockdowns, and the Simpson et al. screen. In each round we compare the predicted effect of the MPE-selected batch with its measured effect. We vary the candidate pool size by subsampling Horlbeck from about $10^3$ to $10^5$ pairs, and we split the error into the additive and interaction branches.
The diagnostic succeeds if the selected-batch optimism grows roughly like $\sqrt{\ln N}$ and concentrates in pairs with low gene support. It fails if the optimism stays flat, in which case the premise of this proposal is wrong and we report that.
The second part is the correction. Within each round we hold out a fold of the already-measured pairs and run MPE on the remaining folds, using a candidate pool size matched to the real one. We then record predicted-minus-realized effects on the pairs that would have been selected. Repeating this across folds gives training data for $\lambda$, which we fit as a small monotone function of $n_i$, $n_j$ and $\ln N$. From the same pseudo-round residuals we build an interval on selected pairs by applying split-conformal quantiles (a distribution-free way to set interval width from held-out errors) to selection-conditional residuals. Acquisition then ranks pairs by the shrunk prediction, optionally plus a small multiple of the interval width.
This idea borrows from two places. PROBECAL (Uncertainty Calibration for Tool-Using Language Agents) fits a small post-hoc calibrator from a few hundred labeled instances and reduces TabMWP calibration error from 0.690 to 0.035. We reuse that cheap post-hoc design, but we condition on selection instead of on hidden states. AERA (AERA: Adaptive Evidence Residual Allocation for Efficient Test-Time Reasoning) shows that confidence signals computed from the evidence itself, such as agreement and entropy, are unreliable predictors of what more effort will yield. That constrains our design: we deliberately supervise $\lambda$ with realized outcomes from pseudo-rounds rather than with the model's own variance.
Baselines
The naive reference point is uniform random sampling of pairs at the same batch size and number of rounds. NAIAD reports roughly twice as many strong perturbations as uniform sampling by round four (Active Learning for Efficient Discovery of Optimal Combinatorial Perturbations).
The strongest existing method is NAIAD itself with MPE acquisition. It reports up to 40% lower root mean square error than GEARS and RECOVER in the low-data regime, plus the twofold hit gain over uniform sampling. We run it with the identical architecture, adaptive embedding schedule, initial seed set, batch size and rounds. Our method changes only the scoring of candidates, so model capacity and compute are held fixed.
A generic uncertainty-aware competitor is a five-member deep ensemble of NAIAD with an upper-confidence-bound rule, which adds a multiple of the ensemble spread to each prediction. It gets the same total tuning budget as our method, searching over the spread multiplier where we search over the form of $\lambda$.
A Bayesian additive baseline uses Bayesian linear regression for the additive part, with a Gaussian-process interaction term over gene embeddings restricted to tractable subsampled pools, and selects by Thompson sampling.
For context on the agent side, BioDiscoveryAgent (BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments) reports outperforming random sampling by 130% on average on the Horlbeck two-gene task. We cite that figure rather than rerun the agent.
Two ablations isolate the key ingredient. The first applies one constant shrinkage factor to the interaction branch, tuned on the same pseudo-rounds. The second fits $\lambda$ on random held-out pairs instead of on selected ones.
Beating NAIAD-MPE means more true top-decile hits in the first four rounds, averaged over at least ten seeds, with confidence intervals that exclude zero on at least two of the three datasets. Beating the ensemble means doing so at one-fifth of its training cost, or matching it at equal cost while lowering selected-batch optimism.
No reported result was found on Lacuna for selection-conditional calibration error, or for any interval coverage, on Norman, Horlbeck or Simpson.
Why not just…?
Why not just use a deep ensemble with an upper-confidence-bound rule? This is the real threat. Ensembles estimate how much the members disagree. They do not directly estimate the bias induced by taking a maximum over $N$ candidates, and the ensemble members share a training set, so their disagreement can be small exactly where the shared data are thin. The head-to-head comparison above settles the question. If the ensemble matches our hit yield and selected-batch calibration at equal cost, we conclude that selection-aware shrinkage is unnecessary and report it that way.
Why not just use a Bayesian linear model or Gaussian process, whose uncertainty is principled? NAIAD's own finding that simple linear models are hard to beat in low data cuts in our favor here. The additive part is not where we expect the optimism. A purely additive model has no interaction term, so it cannot rank synergistic pairs above additive ones. Gaussian processes do model interactions, but they scale cubically with sample size and must be run on subsampled pools. The baseline tests whether principled posteriors on a smaller pool beat corrected point predictions on the full pool.
Why not just shrink the interaction branch by one global constant, like ordinary temperature scaling? If the optimism really is $\sigma\sqrt{2\ln N}$ with $\sigma$ depending on gene support, a constant over-shrinks well-supported pairs and under-shrinks pairs built from rarely measured genes. The constant-shrinkage ablation tests this directly. If it performs as well as the support-dependent $\lambda$, the contribution reduces to "calibrate on selected batches," which the second ablation then isolates.
Why not let a language-model agent with a critic choose the pairs? Verbal self-assessment is fragile and poorly calibrated (Large Language Models Must Be Taught to Know What They Don't Know). Agents reviewing their own finished proposals overestimate success: they predicted 61–77% success where actual success was 22–35% (Agentic Uncertainty Reveals Agentic Overconfidence). BioDiscoveryAgent also reported fixation on narrow literature keywords. An agent picking from a huge pool faces the same winner's curse with no mechanism to measure it, so we treat agents as context rather than as a fix.
Proposed Experiments
The decisive check comes first. On Horlbeck, we plot selected-batch optimism against $\ln N$ across pool sizes, separately for the additive and interaction branches and stratified by gene support. A flat curve falsifies the premise.
Second, in retrospective active-learning replays on all three datasets, we compare cumulative top-decile hits per round with the baselines above. Under the same protocol we measure selected-batch optimism and the coverage of nominal 80% intervals on selected pairs.
Third, on Norman, which has transcriptome-wide readouts, we check whether shrinkage distorts the classification of interaction types such as synergy, buffering and suppression. GEARS (Predicting transcriptional outcomes of novel multigene perturbations with GEARS) serves as the reference there, using its scalar summary of the predicted profile.
The proposal succeeds if three things hold. The optimism scales with pool size as predicted. The support-dependent $\lambda$ beats both ablations on hit yield. Coverage on selected batches approaches nominal while NAIAD-MPE's intervals, built from random held-out residuals, under-cover.
References
- Active Learning for Efficient Discovery of Optimal Combinatorial Perturbations
- Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data
- CALICO: Confident Active Learning with Integrated Calibration
- Uncertainty Calibration for Tool-Using Language Agents
- AERA: Adaptive Evidence Residual Allocation for Efficient Test-Time Reasoning
- BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments
- Large Language Models Must Be Taught to Know What They Don’t Know
- Agentic Uncertainty Reveals Agentic Overconfidence
- Predicting transcriptional outcomes of novel multigene perturbations with GEARS