Practitioners often pick among a handful of language models, checkpoints, or prompts using a validation set of a few dozen labeled items. Two different errors inflate the score they then report for the winner. The first is the winner's curse: whichever candidate scores highest on a small sample is usually the one that got lucky on that sample. The second is contamination: some candidates have seen the validation items, or items very like them, during training. They score well by recall rather than skill. These errors do not simply add. A contaminated candidate has an inflated score, so it is more likely to be selected, and its reported score is then inflated twice.
This proposal builds one resampling procedure that corrects for both errors together. Each validation item is scored twice for every candidate: once in its published form and once in a freshly perturbed form, meaning a rephrasing or, where possible, a newly generated instance of the same problem. The method then resamples these item pairs with replacement and reruns the entire selection pipeline inside every resample, including the step that decides which candidates look contaminated. The output is a small set of candidates that plausibly have the best clean skill, together with an estimate of the chosen candidate's clean score that is corrected both for luck and for leakage.
Core Claim
Existing tools each handle one error. The Leaderboard Lottery resamples benchmark prompts to build confidence sets for the best model, a technique called paired-bootstrap Multiple Comparisons with the Best (MCB). It restores 95% coverage of the true best model where the naive top-of-list model was correct only 0–58% of the time. However, it assumes the prompts are clean and exchangeable, so resampling memorized items simply reproduces the memorization advantage. Contamination detectors such as ConStat and PaCoST audit models on full benchmarks and return a verdict. They do not select anything, and they are not built for 32-item validation sets.
Our claim is that at small validation sizes, contamination screening and selection must share one resampling loop. A per-candidate contamination test on 32 items has little statistical power. If that test is used as a hard filter, its errors silently become selection errors. If the screening is instead redone inside each bootstrap resample, its uncertainty flows into the final confidence set and into the optimism correction.
The target quantity for candidate $m$ is its clean accuracy $\theta_m$, meaning its expected score on fresh items from the task distribution. The estimate combines the two views of each item. Let $\bar s^{\text{orig}}_m$ and $\bar s^{\text{pert}}_m$ be candidate $m$'s mean scores on original and perturbed items, and let $\hat\delta_m = \bar s^{\text{orig}}_m - \bar s^{\text{pert}}_m$ be its observed leakage gap. The clean-score estimate is
$$\hat\theta_m = \bar s^{\text{pert}}_m + w_m\,(\bar s^{\text{orig}}_m - \bar s^{\text{pert}}_m - \hat\delta_m^{+}),$$
where $\hat\delta_m^{+}$ is the gap after shrinkage toward zero, and $w_m \in [0,1]$ weights the original view by how much it can be trusted. For a candidate with no evidence of leakage, the two views are exchangeable measurements, so pooling them reduces variance. For a leaking candidate, the original view contributes little.
Selection picks the candidate with the largest $\hat\theta_m$. Following the optimism-correction loop from clinical prediction work, each bootstrap resample $b$ reruns this whole pipeline and records how much the resample's winner is overestimated on the full sample. The reported clean score for the chosen candidate is its $\hat\theta$ minus the average of that overestimate.
Research Program
The first component is the perturbation source. Rephrasings are cheap and apply to any task, but they may carry the leak with them. The title and framing of Rethinking Benchmark and Contamination for Language Models with Rephrased Samples point to exactly this concern. Where a task allows it, we therefore also use newly generated instances from algorithmic generators in the style of BeyondBench. BeyondBench builds instances with formal solvers, so leaked surface forms cannot help. We treat the gap between rephrasing and regeneration as an object of study in its own right.
The second component is the scoring signal. DataDecide shows that the probability a model assigns to the correct answer gives a smoother ranking signal than accuracy at small scale. Algorithmic Accountability in Small Data shows that discrete metrics jump sharply below about 100 samples. We therefore use correct-answer probability wherever models expose their token probabilities, and accuracy only for text-only APIs. This also sidesteps PaCoST's requirement of token-probability access, because our leakage gap can be computed from outputs alone.
The third component is the resampling unit and the statistical guarantee. We resample items, keeping each item's original and perturbed views together. When there are several perturbations per item, we also resample perturbations within each item, so that noise from rephrasing is counted. Inside each resample we recompute the shrunk gaps, the weights, the clean scores, and a step-down MCB confidence set as in the Leaderboard Lottery.
The final component is analysis. We will characterize when the joint procedure beats simpler pipelines, as a function of validation size, the fraction of candidates that leak, and the size of the leak. We will also connect these results to power limits on detecting contamination, which When Is Benchmark Contamination Detectable? studies directly.
Baselines
The do-nothing reference is naive selection: pick the argmax of original-item accuracy on the $n$ validation items and report that score. True Few-Shot Learning with Language Models supplies the classical small-sample alternatives. On LAMA, leave-one-out cross-validation and minimum description length fall 5–7% absolute behind large-validation selection. They also carry about a 27% chance of choosing a prompt 13% worse than the mean. We rerun both on identical items and candidates.
The strongest existing method for honest selection is paired-bootstrap MCB from The Leaderboard Lottery. It reports at least 95% coverage of the true best model and 95% confidence sets of 8 to 35 models on Open LLM Leaderboard v2 tasks, on the assumption of clean prompts. We run it with the same number of bootstrap draws (B = 2,000) on original items. Our method beats it if two things hold under injected contamination: we keep 95% coverage of the clean-best candidate where MCB falls below it, and we do so without enlarging the set by more than one candidate at the same $n$.
The strongest two-stage competitor is detect-then-select. First apply a ConStat-style filter, which in its paper detects 98.4% of sample-specific contamination against at most 71% for earlier baselines, on full benchmarks with 20 reference models. Then run MCB on the remaining candidates. We give it the same reference models and the same perturbed items, with its threshold tuned on held-out tasks.
We also run cross-prior smoothing from the Algorithmic Accountability paper as a variance-reduction baseline. It uses a larger reference cohort to smooth small-sample metric estimates.
The ablation that isolates our key ingredient is MCB run on perturbed items only. It uses the same perturbations and compute budget but has no pooling of views and no screening inside the loop.
Why not just…?
Why not just evaluate on a contamination-limited benchmark such as LiveBench or MMLU-CF? Where such a benchmark covers the task, it should be used. That is exactly why we use LiveBench and MMLU-CF as ground truth. The target setting, however, is a practitioner's own task with a few dozen labels. A fresh benchmark for that task rarely exists. Even when it does, a small sample from it still suffers the winner's curse. The Leaderboard Lottery shows this on GPQA, where the top model was the true best 0% of the time.
Why not just score only the perturbed items and run standard MCB? This is the real threat. It is our central ablation, and if it matches our method, the honest conclusion is "rephrase, then bootstrap." Our method should help in two places. The first is when most candidates are clean, because pooling both views then roughly doubles the measurements per item. The second is when rephrasings carry part of the leak, because the original-versus-perturbed gap then still carries signal. We will report the range of validation sizes and leak fractions where the ablation is as good as the full method.
Why not just filter with ConStat or PaCoST and then bootstrap? At $n = 32$, per-candidate detection has low power. Both detectors were validated on full benchmarks, and On The Fragility of Benchmark Contamination Detection in Reasoning Models suggests detection is brittle even there. A hard filter that errs does not show up in the confidence set. Our coverage experiment directly measures how often detect-then-select misses the clean-best candidate.
Why not just collect more labels? The Algorithmic Accountability paper puts the threshold for stable metrics near $n \ge 100$. Collecting more labels fixes variance but not contamination: a larger leaked set leaks just as much. We include $n = 128$ to show where variance stops being the main source of error and leakage takes over.
Proposed Experiments
Candidate pools include open model families audited by PaCoST and ConStat (Llama, Mistral, Qwen, Phi-3), intermediate training checkpoints, and prompt sets in the true few-shot LAMA setting. We use controlled contamination in the style of ConStat: we fine-tune some candidates on a known fraction of the validation items, which gives exact ground truth for which candidates leak.
Validation sets of $n \in \{16, 32, 64, 128\}$ items are drawn from MMLU, ARC, and GSM8K, where the file's sources document real leakage: ConStat reports a gap of about 6.6% for Llama-3-70b on ARC and about 8.3% for Mistral-7b on GSM8k. Clean ground truth comes from large pools on LiveBench, MMLU-CF, and BeyondBench generators.
The metrics are coverage of the clean-best candidate, confidence-set size, the clean-score shortfall of the chosen candidate relative to the true best, and the bias and mean squared error of the reported clean score. No reported result was found on Lacuna for any existing method's coverage at these small sizes under contamination.
The decisive checks are threefold. First, under zero contamination, our method must match MCB's coverage and produce equal or smaller sets. Second, as the leak fraction rises, naive selection and MCB should lose coverage while ours holds 95%. Third, the corrected score should track the clean score within its bootstrap interval. If the perturbed-only ablation matches the full method across these conditions, we will report that as the main finding.
References
- The Leaderboard Lottery: Why Top LLM Rankings Are Often Coin Flips, and How to Fix Them
- ConStat: Performance-Based Contamination Detection in Large Language Models
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
- BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
- DataDecide: How to Predict Best Pretraining Data with Small Experiments
- Algorithmic Accountability in Small Data: Sample-Size-Induced Bias Within Classification Metrics
- True Few-Shot Learning with Language Models
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
- On The Fragility of Benchmark Contamination Detection in Reasoning Models