When a genetic association study flags a region of the genome linked to a disease, the variant responsible usually sits outside any gene. It is buried among dozens to thousands of statistically indistinguishable neighbours, and a lab can afford to test only a handful of them. Today's variant scorers, from evolution-based tools like CADD to deep sequence models like Enformer and AlphaGenome, output one number per variant. Nothing tells the experimentalist how many top-ranked candidates they must test to be, say, 90% sure the true one is among them. We propose to answer that question directly. For each locus, we build a conformal candidate set: a short list of variants that is guaranteed to contain the causal variant with a chosen probability. Conformal prediction is a statistical wrapper that turns any model's scores into sets with a finite-sample coverage guarantee, assuming only that calibration and test loci are exchangeable. We then make those lists as short as possible by fusing complementary models in a way that depends on genomic context.
The core claim is that the unit of prediction should be the locus, not the variant, and that the useful quantity is set size at guaranteed coverage, not AUPRC. Framed this way, the question of which model to trust becomes measurable. We can ask how many variants must be tested near promoters versus in distal enhancers, and for Mendelian versus complex traits. We can also ask whether combining alignment-based and sequence-to-function models actually shortens the list, or only reshuffles a ranking. The file of prior work contains no study that builds conformal sets for regulatory variant effect prediction. The closest work adds heuristic reliability scores to viral variant predictions without coverage guarantees (Evaluating variant effect prediction across viruses; Variant effect prediction with reliability estimation across priority viruses).
Why the locus framing, and why efficiency is the hard part
The TraitGym benchmark (Benchmarking DNA Sequence Models for Causal Regulatory Variant Prediction in Human Genetics) already has the structure we need. It pairs each causal variant with nine decoys matched on chromosome, distance to the nearest transcription start site, and allele frequency. It covers 338 Mendelian variants from OMIM and 1,140 complex-trait variants from UK Biobank. We treat each matched group $G$ as one example whose label is the identity of the causal member. A candidate set is
$$C(G)=\{v\in G:\ s(v,G)\ge \hat\tau_{k(G)}\},$$
where $s(v,G)$ is a within-group conformity score for variant $v$. The threshold $\hat\tau_k$ is set on held-out calibration groups belonging to context stratum $k$, so that the causal variant falls below it in at most a fraction $\alpha$ of those groups. This makes
$$\mathbb{P}\big(\text{causal}(G)\in C(G)\ \big|\ k(G)=k\big)\ \ge\ 1-\alpha$$
hold within each stratum. The strata are trait class crossed with distance-to-TSS bins, and the guarantee is the standard group-conditional ("Mondrian") conformal argument. Exchangeability is asked of groups, not of variants. This sidesteps the strong dependence between variants in the same locus, which is exactly what breaks naive variant-level conformal prediction. We split groups by chromosome to limit dependence across calibration and test, and to limit leakage from models trained on held-out chromosomes.
TraitGym itself motivates the stratification: alignment models lead on Mendelian and complex disease variants, while Enformer and Borzoi do better on complex non-disease traits. A single pooled threshold would therefore over-cover the easy stratum and under-cover the hard one. The file documents this conditional-coverage failure for pooled conformal prediction in general (Conformal Prediction for Deep Classifier via Label Ranking; Class conditional conformal prediction for multiple inputs by p-value aggregation).
Stratifying costs calibration data, and set size will grow. The efficiency question is whether better scores can win that size back. Two findings constrain the design. First, rank-based scores like SAPS shrink ImageNet sets from 20.95 to 2.98, but their gains mostly vanish in small label spaces (Conformal Prediction for Deep Classifier via Label Ranking). A ten-member TraitGym group is a small label space, while a real locus is not. Second, uniform temperature scaling can enlarge conformal sets rather than shrink them (Entropy Reweighted Conformal Classification). This rules out fixing the problem by recalibrating each model's scores alone.
Research Program
We first compute per-variant scores from the models TraitGym evaluates: CADD, GPN-MSA, Evo2, Enformer and Borzoi. We add AlphaGenome variant scores where they can be obtained (AlphaGenome: advancing regulatory variant effect prediction with a unified DNA sequence model), and the GPN-style likelihood ratio (DNA language models are powerful predictors of genome-wide variant effects). Raw scores live on incompatible scales, so each model's scores are converted to within-group ranks and z-scores before fusion.
The fusion model is a small gated combiner, fitted on a training split of groups with a within-group softmax (conditional logit) loss. It weights each model's within-group score by a function of context features: distance to TSS, conservation, and whether the variant overlaps annotated accessible chromatin. This operationalizes the TraitGym observation that model reliability depends on context. As a variant, we also train the combiner with a smoothed set-size loss in the style of rank-based conformal training (Enhancing Trustworthiness of Graph Neural Networks with Rank-Based Conformal Training). That work reports up to 11.28% smaller sets than post-hoc calibration on citation graphs, so we expect a modest effect and will test for one rather than assume it. Fused within-group probabilities are then turned into conformity scores in two ways: the plain probability of the true member, and a SAPS-style rank penalty. Both are calibrated per stratum.
Because ten-member groups understate real loci, we also build a stress test with larger candidate pools. We add further decoys drawn by TraitGym's own matching recipe (chromosome, TSS distance, allele frequency) to reach pools of 50, 200 and 1,000 variants. This construction is ours and will be released with the matching code. It is where rank-based scoring should matter and where set bloat becomes visible.
A second line of analysis asks where set size comes from. We decompose mean set size by stratum and by which model contributes most to the fused score. The output is a map of experimental cost: how many variants must be tested per locus at 90% coverage, for each trait class and region type.
Baselines
The naive reference point is the whole group, which has size 10 and trivially full coverage. Next to it we run top-$k$ by CADD, with a single $k$ chosen on calibration groups to reach 90% coverage. That rule is itself a valid conformal set whose score is the causal variant's rank, and it is what practitioners effectively do today. On the Mendelian subset its strength is known: CADD reaches AUPRC 0.85 on TraitGym Mendelian traits (Benchmarking DNA Sequence Models for Causal Regulatory Variant Prediction in Human Genetics). We found no reported conformal set size or coverage result on Lacuna for any variant effect model on TraitGym or CAGI, so the reference numbers in set-size units will be produced here.
The strongest competitor is plain stacking. This is an unconditional logistic combination of all model scores fitted on the same training groups, followed by one pooled split-conformal calibration with standard adaptive prediction sets. It uses exactly the same inputs, splits and tuning budget as our method, and differs only in having no context gating and no stratified thresholds.
We also run two conformal combination methods from Direction 1. The first averages conformity scores across per-model sets (Conformity Score Averaging for Classification). The second selects among per-model conformal sets with stability control using MinSE and its recalibration variant (Valid Selection among Conformal Sets). That paper shows naive pointwise picking of the smallest set loses coverage, and we include such picking as a deliberately invalid reference.
Three ablations isolate the ingredients: fusion with and without context gating, pooled versus stratified calibration, and APS versus SAPS scoring. All hyperparameters for every method are tuned on the same training split. The same number of repeated chromosome-split draws is used throughout.
Beating the baselines means two things at once. Worst-stratum coverage must reach at least $1-\alpha$ within binomial error, which pooled stacking should fail on complex traits. Mean set size in the complex-trait and distal strata must also be smaller than a stratified version of the stacking baseline, with the difference exceeding split-to-split variation.
Why not just…?
Why not just temperature-scale each model and run standard conformal prediction? Temperature scaling preserves each model's ranking within a group, so the resulting set is the same as a top-$k$ rule on that model. It cannot combine complementary models. Uniform temperature has also been shown to inflate sets (Entropy Reweighted Conformal Classification). We include per-model temperature-scaled sets in the comparison, so this claim is tested rather than assumed.
Why not just stack the models and calibrate once? This is the real threat. Stacking with context features as inputs is close to our gated combiner. If stratified stacking matches context-gated fusion on set size, we will say so and adopt it. The contribution then rests on the locus-level formulation, the stratified guarantee, and the map of experimental cost. What stacking alone does not supply is conditional validity. The pooled-versus-stratified ablation will show whether pooled stacking under-covers complex-trait loci, which the model dichotomy in TraitGym predicts.
Why not just select, per locus, whichever model gives the smallest valid set? Stable selection is principled (Valid Selection among Conformal Sets). However, it chooses among whole models, whereas the evidence suggests different models are right about different variants within the same locus. It also solves a linear program per query, and pays a small inflation of the error rate. We run MinSE directly. If it produces smaller sets than fusion at matched coverage, that result favours selection over fusion and we will report it.
Why not just use ACMG/AMP evidence thresholds or fixed CADD cutoffs? Evidence categories are variant-level and parametric. On the CAGI NAGLU challenge only 6% of variants reached "Strong" and 31% "Supporting" (CAGI, the Critical Assessment of Genome Interpretation, establishes progress and prospects for computational genetic variant interpretation methods). Neither category says how many candidates in a locus must be tested. A fixed cutoff also yields sets of unbounded size with unguaranteed coverage. We report the coverage and size that a CADD PHRED > 20 cutoff achieves within groups, for direct comparison.
Proposed Experiments
The primary experiment runs every method on TraitGym Mendelian and complex-trait groups at $\alpha\in\{0.05,0.1,0.2\}$, over 20 chromosome-split draws. For each stratum we report marginal and stratum-conditional coverage, mean set size, and the fraction of singleton sets.
The first decisive check is whether pooled calibration under-covers complex-trait distal loci while stratified calibration does not. If both cover equally, stratification is unnecessary and we drop it.
The second decisive check is whether fusion shrinks complex-trait sets relative to the best single model at matched stratum coverage. If the fused set is no smaller than top-$k$ Enformer or Borzoi in that stratum, then model complementarity does not translate into cheaper experiments. That would be a useful negative result.
The third check is the expanded-pool stress test. We expect SAPS-style scoring to separate from APS as pools grow from 10 to 1,000, consistent with the label-space dependence reported by Conformal Prediction for Deep Classifier via Label Ranking.
Finally, as an out-of-distribution sanity check, we calibrate on TraitGym and evaluate coverage on the CAGI5 regulatory MPRA challenge, reframed as within-element candidate sets. A coverage drop there would quantify how far the exchangeability assumption can be stretched across data sources. It would also tell us whether calibration must be redone for each assay type.
References
- Evaluating variant effect prediction across viruses
- Variant effect prediction with reliability estimation across priority viruses
- Benchmarking DNA Sequence Models for Causal Regulatory Variant Prediction in Human Genetics
- Conformal Prediction for Deep Classifier via Label Ranking
- Class conditional conformal prediction for multiple inputs by p-value aggregation
- Entropy Reweighted Conformal Classification
- AlphaGenome: advancing regulatory variant effect prediction with a unified DNA sequence model
- DNA language models are powerful predictors of genome-wide variant effects
- Enhancing Trustworthiness of Graph Neural Networks with Rank-Based Conformal Training
- Conformity Score Averaging for Classification
- Valid Selection among Conformal Sets
- CAGI, the Critical Assessment of Genome Interpretation, establishes progress and prospects for computational genetic variant interpretation methods