Contamination-Robust Leaderboard Winners via Metamorphic Twins

Public leaderboards pick a "best" language model by taking the highest score on a fixed set of test questions. Two separate problems make that pick unreliable. The first is the winner's curse: the top score partly reflects lucky noise in which questions happened to be asked. The second is contamination: some models have seen the test questions during training. Statisticians have built confidence sets for "which models could plausibly be best" that fix the first problem, and contamination researchers have built detectors for the second. No one has combined them. As a result, a statistically careful leaderboard can still certify, with 95% confidence, a set of leaders that consists of the models that memorized the test.

We propose to score every model on metamorphic twins of each benchmark question. A metamorphic twin is a programmatically generated variant whose correct answer follows from a stated rule, either "same answer as the original" or "answer changes in this computable way." The twins are generated fresh for each evaluation round, and every model in that round sees the same twins. We then build the plausible-best set with a bootstrap that resamples at two levels: first the source questions, then the twins within each question. The resulting set is meant to cover the model that would win on uncontaminated versions of the task. It needs only black-box outputs and no trusted clean reference models.

Core Claim

The quantity a leaderboard should rank models by is each model's expected accuracy over the family of valid variants of each source item, which we call its metamorphic-closure accuracy. This differs from accuracy on the literal items. We claim three things:

  1. Paired-bootstrap confidence sets computed on the literal items lose coverage of the truly best clean model once contamination differs across models.
  2. Confidence sets computed on twins with resampling at both the item level and the twin level restore that coverage.
  3. They do so without becoming as uninformative as the near-all-model sets seen in pairwise-preference settings.

Two further points about the twins matter:

  • Directional relations. Relations whose answer must change by a predictable amount, such as altering a number in a math word problem, are what defeat memorization. Pure paraphrase invariance only partly does, because a memorizing model can map a near-copy back to a stored answer.
  • Two-level resampling. This is necessary for validity. Without it, the confidence set is honest only about which questions were drawn and not about which twins were generated. Since a new round regenerates the twins, that is the wrong guarantee.

Prior Work and the Gap

Three lines of work motivate this proposal.

Post-selection inference for leaderboards. The Leaderboard Lottery treats leaderboard ranking as inference after selection and finds the following:

  • In simulations calibrated to the Open LLM Leaderboard v2, the empirical winner matched the true best 0% of the time on GPQA and 0–58% on the other five tasks.
  • Paired-bootstrap Multiple Comparisons with the Best (MCB), with Holm step-down correction, restores at least 95% coverage, with plausible-leader sets of 8 to 35 models.

It assumes independent, uncontaminated items. We keep its MCB machinery and change both the items it resamples and the estimand it targets.

Contamination detectors that compare originals with rephrasings. ConStat measures the gap between a model's score on originals and its score on rephrased or synthetic items, using 20 trusted reference models. It reports 98.4% detection against 71% for prior baselines, and contamination inflation above 10% for the top-3 Open LLM Leaderboard models of its time. PaCoST runs a paired t-test on a model's self-reported confidence ("is this answer correct?") for originals versus rephrasings.

Both treat contamination as a per-model diagnostic. Neither propagates that uncertainty into a ranking. ConStat breaks if the reference models are themselves contaminated, and PaCoST needs log-probabilities. We borrow their core insight, that the original-versus-variant gap reveals memorization, and drop both dependencies.

Formal metamorphic relations. Work on metamorphic testing, which checks a system without a known correct output by asserting relations between outputs on related inputs, supplies the generators:

All of these evaluate one model in isolation. None asks whether differences in pass rates across many models survive selection.

Dynamic generation and the reproducibility tension. DyVal shows that freshly generated reasoning tasks expose collapse: phi-1.5 and WizardMath-13B score near zero despite strong static scores, and GPT-4 loses 13.6% when reachability tasks are presented in reversed order. Fresh instances, however, break the fixed shared sample that paired leaderboard comparisons need, and DyVal is limited to formal domains. Our seeded per-round twin generation keeps the pairing, because all models in a round see identical twins, while preventing the twins themselves from entering future training corpora.

Constraints from prior work. Two papers shape how we design the bootstrap rather than motivate the problem:

  • Expected Validation Performance shows that an unbiased but high-variance estimator of the maximum led to more wrong conclusions than a slightly biased one on SST-5. We should therefore not chase unbiasedness at any cost to set size.
  • BBC-F shows that resampling coarser units (cross-validation folds rather than instances) can keep 95% coverage at a fraction of the cost. This is a precedent for our choice of resampling unit.

Research Program

Twin generators

We will build relation libraries for three benchmarks: MMLU-Pro, GSM8k, and ARC-Challenge (Think you have Solved Question Answering? Try ARC). As a formal-domain check, we will also reuse the DyVal generators.

Invariance relations (the answer must not change):

  • Permuting answer options.
  • Replacing distractors with new plausible wrong options.
  • Swapping entities and surface details in the CheckList style.
  • Paraphrasing with an answer-preservation verifier.

Directional relations (the answer must change in a known way):

  • For GSM8k, each solution is parsed into an executable template. Changing the numbers yields a new, computable answer.
  • For multiple-choice items, the question is inverted to "which option is NOT correct," with a known answer set.
  • For ARC-Challenge, a premise is negated where the gold answer flips in a known way.

Quality control. Every relation passes an audit before use. Annotators label a random 200 twins per relation for answer validity, and any relation with more than 3% label drift is dropped. This operationalizes METAL's perturbation-quality concern as a hard gate rather than a soft weight.

Estimand and inference

Let $i$ index source items, $j$ index twins of item $i$, and $c_{mij} \in \{0,1\}$ record whether model $m$ answers twin $j$ of item $i$ correctly. Model $m$'s metamorphic-closure accuracy is

$$\theta_m = \mathbb{E}_i \, \mathbb{E}_{j \mid i} \, [c_{mij}],$$

where the outer expectation is over source items and the inner one is over the twin distribution that the relation library induces for that item.

Confidence sets come from paired-bootstrap MCB with Holm correction, following The Leaderboard Lottery. The difference is the resampling: we draw items with replacement, then twins with replacement within each drawn item, and apply the same draw to every model. The per-model gap between literal-item accuracy and $\theta_m$ is reported as a black-box contamination estimate with its own bootstrap interval. It is used for diagnosis only; the ranking uses $\theta_m$ alone.

Seeded release protocol

Each round, the organizer commits to a hash of the generator seed and then reveals the seed after evaluation. This keeps the pipeline standardized and reproducible after the fact, while ensuring the twins cannot have been trained on beforehand.

Baselines

Naive reference: marginal intervals. Score each model on the literal items and declare a winner when its standard 95% interval does not overlap the others. This is how leaderboards are read today. The Leaderboard Lottery reports 0% true-best recovery on GPQA and 0–58% on the other tasks under clean simulation. We expect it to do worse under contamination. Our method beats it if its coverage of the true clean best is at least 95% where the marginal approach falls below that.

Strongest existing inference method: literal-item MCB. Run paired-bootstrap MCB with Holm correction on the literal items, as in The Leaderboard Lottery. Its reported result on clean simulated Open LLM Leaderboard v2 tasks is at least 95% coverage of the true best with sets of 8 to 35 models. That paper reports no result under contamination, and we found none on Lacuna for contamination-aware leaderboard coverage. It will be run on the same models, items, and number of bootstrap replicates as our method, so the only difference is the resampled units and the estimand. Beating it means holding at least 95% coverage of the clean best under injected contamination where it does not, and with a set no more than 1.5 times its clean-condition size.

Strongest contamination-aware composite: ConStat plus MCB. Subtract ConStat's estimated inflation $\delta$ from each model's literal score, then run MCB. This is the obvious way to bolt existing tools together. It gets the same compute budget, and we give it its best case: all 20 reference-model families from ConStat where available, and GPT-4-class rephrasings. We also run it in a stress condition where the reference models themselves receive injected contamination. Our method beats it if it matches its coverage in the clean-reference condition and keeps coverage in the contaminated-reference condition where the composite loses it.

Ablations isolating the key ingredients: - Single-level bootstrap on twins, which tests whether twin-level resampling matters. - Invariance-only twins, which tests whether directional relations matter. - Unaudited LLM paraphrases in place of audited relations, which tests whether formal, verified relations matter compared with the ad hoc rephrasing used by ConStat and PaCoST.

Why not just…?

Why not just plug ConStat's contamination estimate into MCB? This is the most serious competitor, and in the favorable case it may well match us. Its weaknesses are structural:

  • It needs clean reference models. As contaminated web corpora like FineWeb become the common pretraining substrate, that assumption erodes.
  • Subtracting a point estimate of $\delta$ ignores the uncertainty in $\delta$, so the corrected set inherits a hidden bias.

The contaminated-reference stress test settles this. If ConStat plus MCB keeps coverage there too, our advantage reduces to black-box simplicity, and we will report it that way.

Why not just evaluate on freshly generated problems, as DyVal does? On formal reasoning tasks this is a real alternative, and we include DyVal tasks precisely to compare against it. Its limitations are that fresh synthetic problems define a different task from the benchmark being ranked, and that DyVal does not cover knowledge-heavy multiple-choice benchmarks like MMLU-Pro. Twins anchored to source items keep the content of the original benchmark while removing the memorized surface form. If rankings from DyVal-style fresh tasks and from twins agree on the formal tasks, that supports twins as the general-purpose version.

Why not just bootstrap over the twins as ordinary independent items? This is the cheapest fix and the one we most need to rule out. Twins of the same item are correlated, so treating them as independent shrinks the intervals and overstates certainty about a regenerated round. The single-level ablation tests exactly this, measuring coverage across repeated regenerations of the twins. If it holds 95% coverage anyway, the two-level design is unnecessary and we will drop it.

Why not just use a harder static benchmark? MMLU-Pro widened a 1% gap on MMLU to 9% by moving to 10 options, but it is a static file and will leak like its predecessor. Difficulty addresses saturation, not memorization.

Proposed Experiments

Controlled contamination study

This study supplies ground truth. We take about 24 open models across several families and sizes. For a random subset of them, we fine-tune on literal test items at controlled rates: 0, 10, 30, and 100% of items, with answer-only leakage as a separate condition following PaCoST's finding that string matching misses it. Clean ground-truth accuracy for each model comes from a large held-out pool of audited twins that is never used for ranking. That pool defines the true clean best.

Each condition is repeated over 200 resamplings of items and fresh twin seeds. We measure:

  • Coverage of the true clean best.
  • Average set size.
  • How often contaminated models enter the set while the clean best is excluded.
  • Rank correlation between the reported ranking and the clean ranking.

Real-model audit

We will run about 40 public open models on MMLU-Pro, GSM8k, and ARC-Challenge. We will report how the plausible-leader sets change between literal-item MCB and twin MCB, and compare our black-box contamination gaps with ConStat's published gaps for overlapping models: Llama-3-70b on ARC (about 6.6%) and Mistral-7b-v0.1 on GSM8k (about 8.3%).

Decisive checks

  1. The problem must be real. In the controlled study, literal-item MCB must lose coverage of the clean best once contamination differs across models. If it does not, the problem is smaller than claimed.
  2. Twin MCB must restore coverage at an acceptable cost. Twin MCB with two-level resampling must restore at least 95% coverage with a set no larger than 1.5 times clean-condition MCB. Otherwise the method is valid but too uninformative to use, which is the failure mode seen in pairwise-preference rankings.
  3. The ingredients must each matter. Removing directional relations or the twin-level resampling should measurably reduce coverage. If neither ablation hurts, the contribution is simply "rank on audited variants," and the note's claims shrink accordingly.