Ranking Which Queries Deserve an Ensemble When the Training Logs Are Contaminated

A system that serves a language model can answer each query cheaply with one small model, or expensively by escalating to a larger model or an ensemble (several models whose answers are combined). It only has budget to escalate a fraction of queries, so what it needs is an ordering: which queries gain the most from escalation. This is a treatment-effect ranking problem. The "treatment" is escalation, and the quantity to rank is each query's expected accuracy gain from it. The data used to learn that ranking usually comes from benchmark runs or logged outcomes, and that data has a known flaw. Small models often "solve" benchmark items they have memorized from pretraining, and their success on those items disappears when the problem is reworded or its numbers change. A router (the model that decides which queries to escalate) trained on such logs learns that memorized items are easy for the cheap model. It therefore under-escalates the very query types where escalation helps most on fresh inputs.

We propose to learn the escalation ranking directly, using two kinds of data. The first is a large, cheap pool of outcomes on original benchmark items, which are biased by contamination. The second is a small, expensive pool of outcomes on perturbed variants of the same items, which are not. Cheap contamination probes, computed only at training time, predict where the two pools disagree. The bias-corrected labels are then fed into a pairwise ranking objective that is insensitive to errors in the bias model. The deployed router reads only the query, so all contamination handling happens offline. This addresses the cost problem in which live contamination filtering consumed 83–95% of system spend in Allocation, Not Volume.

Core Claim

Define a query $x$, escalation $T\in\{0,1\}$, and the clean outcome $Y^{c}(T)$, meaning correctness on a perturbed variant of $x$ that the model cannot have memorized. The deployment-relevant quantity is the clean lift

$$\tau(x)=\mathbb{E}[Y^{c}(1)-Y^{c}(0)\mid x].$$

Static benchmark logs instead give the static outcome $Y^{s}(T)$, measured on the original wording. Their lift is shifted by a contamination bias

$$b(x)=\mathbb{E}[(Y^{s}(1)-Y^{s}(0))-(Y^{c}(1)-Y^{c}(0))\mid x].$$

This bias is concentrated on items the cheap model has memorized. It is usually negative, because memorization inflates $Y^{s}(0)$.

Our claim has three parts. First, $b(x)$ is large and uneven across items, so routers trained on static lift misrank escalation in a measurable way. Second, $b$ is predictable from probe features $z(x)$ that are cheap to compute offline. The probes we use are the rate at which accuracy collapses as sampling temperature rises, which was reported to be up to 40x faster for contaminated models in Quantifying the Effect of Test Set Contamination on Generative Evaluations, the item's token-level surprise, and anomalous item-response patterns of the kind flagged by the item-response-theory guessing parameter in Efficiently Measuring the Cognitive Ability of LLMs. Third, combining abundant static data with a small clean subset through a direct ranking objective yields a better escalation ordering than using either pool alone.

Relation to Prior Work

The closest work is Meta-Router. It trains a two-model router from scarce gold-standard labels plus abundant biased judge labels, models the query-dependent bias as a conditional treatment effect, and removes it with R- and doubly-robust regression learners. We borrow its two-source structure. We differ in three ways. Our bias comes from memorization, not judge preference, and it has a measurable signature that our probes target. We optimize the ranking of lift rather than regressing a pointwise shift. And we evaluate against ground-truth contamination doses that we inject ourselves.

The case for ranking over regression comes from the treatment-effect literature. Rankability-enhanced Revenue Uplift Modeling shows that squared error is only a loose bound on ranking error. Learning to Rank for Optimal Treatment Allocation beats DR-Learner in up to 24 of 30 replications at $N=100$–$250$. Rank-Learner supplies the method we reuse: cross-fitted nuisance models followed by a Neyman-orthogonal pairwise ranking loss, meaning a loss whose value is first-order unaffected by small errors in those nuisance models. In our setting the nuisances are the bias model and the probability that an item was selected for clean measurement.

Rank-Learner also constrains our design. It reports degradation when nuisances are badly misspecified and sensitivity to its smoothness parameter $\kappa$. We therefore tune $\kappa$ on clean-subset AUTOC (area under the targeting operator characteristic, which scores how much lift is captured as one escalates down the ranking). We also stress-test the method with deliberately weak bias models.

Two lines of work motivate the problem. The contamination studies (Quantifying…, Causally Quantifying…) show that contaminated small models reach near-perfect MATH scores, collapse on rephrased items, and become more confident as they generate ("deterministic lock-in"). DyVal shows that models strong on static benchmarks fall to near zero on procedurally generated problems. None of these papers asks what contamination does to a downstream allocation policy.

On the systems side, Allocation, Not Volume allocates compute using heuristic self-uncertainty. Beyond Gemini-3-Pro routes by similarity to an "embedding bank" of past queries labelled with whether they were answered correctly. Both depend on historical correctness labels that contamination can corrupt.

Research Program

The first step builds a testbed in which contamination is known item by item. Following the injection design of the MATH contamination papers, we pretrain small cheap models on FineWeb-Edu with replicas of a random subset of MATH test items injected at varied doses, including zero. Every item then has a known dose. The escalation arm is majority voting over samples from a pool of larger open models. For every item we generate perturbed variants by rephrasing and re-numbering, and we record outcomes for both arms on the original item and on its variants. Because both arms are run on every item, true clean lift is directly measurable for evaluation.

The second step trains the learner. Clean outcomes are revealed for only a small subset of $n\in\{100,250,1000\}$ items, mirroring Meta-Router's scarce-gold regime. Static outcomes are available everywhere. A cross-fitted bias model maps probes $z(x)$ to $\hat b$. Pseudo-labels are formed as static lift minus $\hat b$, with an orthogonal correction term computed on the clean subset. The ranker takes query features only and is trained with Rank-Learner's pairwise loss.

The third step moves to natural contamination. We use off-the-shelf small open models as the cheap arm. The clean targets are perturbed MATH variants and freshly generated DyVal items at depths D1–D4. We check whether static-trained routers misallocate in this setting too. As an external check on whether the approach transfers beyond deterministic math, we use the ForecastBench subset from Allocation, Not Volume, where clean outcomes come from questions dated after each model's knowledge cutoff rather than from perturbations.

Baselines

The naive reference points are never escalating, always escalating, and escalating a random fraction. Together these fix the two ends of the cost–accuracy curve and the line between them.

Confidence thresholding escalates the queries on which the cheap model's answer entropy is highest. It is the standard cheap router. We run it on the same queries with the threshold swept to match each budget.

The static-data router is our proposed learner trained only on static lift with no correction. Comparing against it isolates the harm that contamination does.

The strongest existing competitor is Meta-Router's approach transplanted to this setting: a DR-learner fit on the static pool, debiased with the clean subset, with queries then sorted by predicted lift. It was designed for exactly this two-source problem, so beating it is the meaningful test. It receives the same clean subset, the same features, the same cross-fitting folds, and the same hyperparameter search budget. Meta-Router reports its gains only in a figure, so no numeric result for it was found on Lacuna. For a heuristic allocator, Allocation, Not Volume reports 80.7% accuracy at \$0.18 per question against 78.1% at \$0.90 for the best static ensemble on its 228-question ForecastBench subset (gpt-5.4-mini). We will reimplement its uncertainty-driven escalation rule on our arms. No reported result was found on Lacuna for escalation routing on MATH or DyVal, so the numbers on those benchmarks will be ours.

We also include a clean-only ranker trained only on the $n$ clean items, which uses no static data at all.

Three ablations isolate our ingredients. The first removes the probes, so the bias is modelled from query features alone. The second replaces the ranking loss with regression on the same corrected labels. The third replaces the learned $\hat b$ with the true injected dose, which gives an oracle upper bound.

Beating the baselines means higher clean AUTOC and higher clean accuracy at 10%, 20%, and 30% escalation budgets than both the debiased DR-learner and the clean-only ranker. The gains must hold across seeds with paired bootstrap intervals that exclude zero, at $n\le 250$.

Why not just…?

Why not just train the router on perturbed variants only? This is the most serious threat. If perturbing every training item is cheap, as it is for MATH, then the clean-only ranker with large $n$ may simply win, and the static pool would be unnecessary. We therefore treat the clean-only ranker's learning curve as the yardstick. The proposal is worthwhile only where clean labels are scarce: open-ended tasks that DyVal notes resist procedural generation, or temporal-cutoff data that accumulates slowly. We will report the clean $n$ at which the clean-only ranker matches ours. If that crossover sits at a few hundred items even on forecasting, we will state that the static pool adds little.

Why not just escalate when the cheap model is unsure? Lock-in makes contaminated models more confident along memorized paths (Quantifying…). So entropy should be lowest exactly on the items whose clean lift is highest. The testbed with injected doses measures this directly: we plot escalation rate against dose for each router.

Why not just debias with a DR-learner and sort, as Meta-Router does? It may well perform comparably when clean $n$ is large. The ranking literature predicts our advantage appears at small $n$, where Learning to Rank for Optimal Treatment Allocation found its largest gains. The regression-loss ablation separates the effect of the loss from the effect of the probes.

Why not just run the ensemble on everything? Allocation, Not Volume found that static ensembles cost 5.1–13x more for no better accuracy. Beyond Gemini-3-Pro shows that weak members can drag aggregated answers down. Our always-escalate baseline will show where on the cost curve this option sits.

Proposed Experiments

The decisive check uses the injected testbed. Bin items by known dose. On high-dose items, the static-data router should escalate least while true clean lift is largest. Our learner should close this gap, and its learned $\hat b$ should track the injected dose. If the probes fail to predict $b$, meaning probe ablation gives no loss, then the contamination component adds nothing and the result reduces to Meta-Router with a ranking loss. We would report it as such.

The second check is transfer. On natural contamination, using DyVal and perturbed MATH, the static-data router's AUTOC deficit relative to our learner should persist. On ForecastBench, the ordering of methods should match the ordering on math.

The third check is robustness. We degrade the bias model to a linear fit and track AUTOC, which tests Rank-Learner's reported weakness under misspecification. We also report the sensitivity of results to $\kappa$.

The fourth check is cost. We report total offline probe cost against the online filter cost reported in Allocation, Not Volume, at matched clean accuracy.