Contamination as Recurrent, Distance-Violating Errors: Robust Leaderboards from Response Matrices

A benchmark leaderboard is a noisy sorting problem. Each test item acts as a comparison between two models: on an item one model solves and the other fails, the item "votes" for the first. Standard leaderboards average these votes and treat disagreements as independent coin flips at a uniform rate. Benchmark contamination, where a model saw the test items during training, breaks both halves of that assumption. The resulting errors are recurrent: re-running a model on a leaked item returns the same inflated answer every time, so resampling never averages them away. They are also distance-violating: a weak model that memorized an item beats much stronger models on it, which genuine skill differences almost never produce. This note proposes a leaderboard estimator built on that observation, and a theory of how far contamination can move a model up the ranking before it becomes visible.

The core claim is that the non-uniform structure of honest errors makes contamination identifiable from model outputs alone. Honest disagreements between models concentrate on pairs of nearly equal strength and on items near their difficulty level. Contamination can therefore move a model past close neighbors undetectably, but not past distant ones. This matters because the tools that inspect model internals now fail in the settings people care about. In On The Fragility of Benchmark Contamination Detection in Reasoning Models, ten loss- and likelihood-based detectors fall to 50–58% AUROC once reasoning models are contaminated with chain-of-thought data. Rephrasing also hides the temporal signals that auditors rely on (Test of Time). A detector that works purely at the level of rankings sidesteps both failure modes.

Core Claim

Let $y_{ai}\in\{0,1\}$ record whether model $a$ solves item $i$. Let $\theta_a$ be the model's latent ability and $b_i$ the item's difficulty. The honest part of the data follows a distance-dependent model in which success probability depends on the gap $\theta_a-b_i$. As a result, the chance that a weaker model $a$ beats a stronger model $c$ on a disagreeing item shrinks with $\theta_c-\theta_a$. Contamination adds a sparse set $L_a$ of items on which $y_{ai}=1$ deterministically, whatever the ability gap. The estimator solves

$$\max_{\theta,\,b,\,\{L_a\}} \sum_{a}\sum_{i\notin L_a}\log p(y_{ai}\mid \theta_a-b_i)\quad \text{subject to}\quad |L_a|\le \varepsilon N,$$

where $N$ is the number of items and $\varepsilon$ is a per-model contamination budget. The rank is then read off from $\theta$.

The accompanying conjecture extends Sorting with Recurrent Comparison Errors to this setting. That paper proves recurrent errors force $\Theta(\log n)$ maximum dislocation for uniform flip rate $p$, and it names distance-dependent recurrent errors as an open problem. Here, dislocation means how many positions a model ends up from its true rank. The conjecture is that when errors are recurrent but budgeted, and honest noise decays with distance, each model's dislocation is bounded by the number of models whose true gap lies below a threshold $\delta(\varepsilon)$. Under uniform noise no such local guarantee exists, because a flip against a distant model is exactly as likely as a flip against a neighbor. The same bound yields certified rank intervals: an interval for each model that cannot be escaped with budget $\varepsilon$.

Relation to Prior Work

Two lines of sorting theory supply the method. Sorting with Recurrent Comparison Errors provides the recurrent-error model and Window Sort, which ranks items by win counts computed within shrinking local windows. We reuse its local-window aggregation but replace its uniform flip rate with item-concentrated, budgeted flips. Optimal Bounds for Noisy Sorting shows that separating adjacent elements costs something fundamentally different from coarse sorting. Its analysis assumes independent, uniform flips; we use it only as the conceptual reason that close pairs are where ranking uncertainty lives. Efficient Ranking from Pairwise Comparisons and Complexity of Round-Robin Allocation with Potentially Noisy Queries motivate the distance-dependent noise assumption, since both report errors concentrating on near-equal alternatives.

The closest evaluation work is Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective. It fits item response theory (a psychometric model of item difficulty and discrimination) and notes that contamination produces anomalous response vectors, with hard items solved and easy ones failed. Our proposal differs in two ways. First, it estimates the outlier sets jointly with the calibration. That work calibrates on a model pool that may itself be contaminated, and it flags anomalies afterwards. Second, it targets a ranking guarantee with a stated budget, rather than an anomaly score. The Leaderboard Illusion motivates the ranking focus. It shows that small, selectively exploited variance in close matchups moves Arena ratings by about 100 Elo, and it leaves the underlying noise model as standard Bradley-Terry, a model of pairwise wins with a single global noise scale. Rethinking LLM Evaluation: Can We Evaluate LLMs with 200× Less Data? (EssenceBench) is a constraint. It preserves whatever ranking the historical score matrix encodes, including a contaminated one, so ranking preservation is not the same as ranking correctness.

One cited result partly contradicts the proposal. Chain-of-thought contamination may teach a model a whole problem family rather than individual items (On The Fragility of Benchmark Contamination Detection in Reasoning Models). If so, the leak is no longer sparse and looks like a genuine ability shift. We treat this as a boundary to measure, not to argue away: the method targets item-concentrated leakage, and one experiment below locates where family-level generalization defeats it.

The Mallows-model warning in Properties of the Mallows Model Depending on the Number of Alternatives constrains the metrics. It shows fixed-dispersion ranking models make large pools look artificially easy to rank. We therefore report dislocation normalized by leaderboard size, and we test at several pool sizes.

Research Program

The theory component states and proves the local dislocation bound for a stylized version of the model: logistic distance-dependent honest noise plus adversarial recurrent flips of at most $\varepsilon N$ items per model. It also derives the certified rank intervals. A useful byproduct is a polynomial-time estimator, since Window Sort's quadratic cost is acceptable for hundreds of models even though it would not be for millions of items. The algorithmic component implements the budgeted estimator by alternating two steps: fit abilities and difficulties, then reassign each model's $\varepsilon N$ most distance-inconsistent successes to $L_a$. A Window-Sort variant computes local win counts only on items not in any $L_a$.

The empirical component builds controlled contamination with known ground truth, which is the only way to score a ranking method. For a pool of open models, we create contaminated twins by following the two pipelines of On The Fragility of Benchmark Contamination Detection in Reasoning Models. The first is fine-tuning on leaked items followed by one clean epoch of GRPO reinforcement learning. The second is chain-of-thought contamination of DeepSeek-R1-Distill-Llama-8B. We vary which items leak, what fraction leaks, and which models receive leaks. The reference ranking is the ranking of the clean twins on the same benchmark. A second reference, which checks that the first is sensible, is performance on evolved items from EvoEval.

Baselines

The naive reference point is mean accuracy on the full benchmark, together with its Bradley-Terry equivalent over per-item pairwise outcomes and item-bootstrap confidence intervals. This is what leaderboards report today. It embodies the uniform, independent-noise assumption, and it is computed on exactly the same response matrix our estimator sees. We beat it if normalized dislocation relative to the clean-twin ranking falls substantially. We also require the bootstrap intervals to be shown to under-cover the true rank on contaminated models while our certified intervals do cover it.

The strongest existing method on this question is item response theory with anomaly flagging, as in Efficiently Measuring the Cognitive Ability of LLMs. We run a three-parameter model on the same matrix, calibrated on the full pool including contaminated twins, which is exactly the setting that paper leaves open. We then remove flagged responses and re-rank. We give it the same tuning budget for its flagging threshold that we give our $\varepsilon$, chosen on a held-out contamination configuration. That paper reports rank correlation above 0.90 with full benchmarks at 80% fewer items, but no reported result found on Lacuna for its contamination-detection accuracy or for ranking recovery under contamination. We beat it if we recover the clean ranking better when many models share the same leaked items, the case in which its calibration absorbs the leak into item difficulty.

The strongest model-internal alternative is LiRA membership inference, a detector that compares an item's likelihood under the target model and under reference models. On The Fragility of Benchmark Contamination Detection in Reasoning Models reports 89.1% AUROC after fine-tuning contamination, dropping by about 14% after one epoch of clean GRPO, and 50–58% AUROC across detectors in the chain-of-thought setting. We run LiRA with the same reference models on the same twins, filter the items it flags, and re-rank. For item-level detection, beating it means exceeding its post-GRPO and chain-of-thought AUROC using outputs only.

The key ablation is our estimator with the distance dependence removed: outliers are selected by residual under a uniform flip model rather than by ability gap. If the theory is right, this variant loses the ability to tell contamination from honest near-neighbor disagreement. Its dislocation should approach the naive baseline's.

Why not just…?

Why not just evaluate on fresh or evolved items instead? Fresh items are the gold standard where they exist, and we use them as a reference. But generators like DyVal are restricted to formal tasks such as arithmetic, logic, and graph reachability. Evolved and rephrased items can launder contamination rather than remove it: Test of Time finds the source paper remains the top influence in 17.5% of transformed questions, and perturbation removes LiveCodeBench's post-cutoff decay. Meanwhile, most leaderboards still rank models on static suites. Our method audits those rankings rather than replacing them.

Why not just run membership-inference detectors and drop flagged items? Because the cited evidence says they collapse to near chance after clean reinforcement learning and in chain-of-thought contamination. Our experiment reruns exactly those pipelines to check whether output-level distance violations survive where likelihood gaps do not. If they do not survive either, that result is itself informative: it would mean reinforcement learning erases contamination at the behavioral level too.

Why not just use item response theory with person-fit statistics? This is the real threat. Our estimator is essentially a robust, budgeted version of that model, so the question is empirical. We expect robust joint estimation to matter mainly when leaks are shared across many models, because shared leaks corrupt calibration. When leaks are idiosyncratic to single models, standard item response theory may match us. If it matches us everywhere, the contribution shrinks to the dislocation bound and certified intervals, and we would report that plainly.

Why not just subtract a worst-case margin, treating any accuracy gap below $\varepsilon$ as a tie? That rule is valid but maximally conservative. It ignores that a leak inflating a weak model must show up as wins against far stronger models. We report it as a certification baseline and measure how much narrower our intervals are at equal coverage.

Proposed Experiments

The main experiment uses the reasoning benchmarks from the fragility study: AIME, AMC, GPQA Diamond, OlympiadBench, and Minerva Math. It also uses GSM8K and HumanEval with EvoEval as the clean-variant reference. We sweep leak fraction and the number of models sharing a leak, run both contamination pipelines, and report four things: normalized maximum and total dislocation against clean twins, Kendall correlation between rankings, coverage and width of rank intervals, and item-level AUROC. The false-flag rate on uncontaminated models is tracked throughout, since an estimator that demotes honest models is worse than useless.

The decisive check is the local-leapfrogging prediction. Contaminated models should rise past close neighbors but be caught when the leak would move them past distant ones, and the uniform-noise ablation should show no such asymmetry. A second check looks for a boundary: as contamination moves from verbatim items to chain-of-thought family-level training, sparsity breaks down, and we measure the leak fraction at which recovery fails. Finally, we apply the estimator to real historical score matrices from the EssenceBench Codebase at several pool sizes. We report which published rank orderings fall inside certified intervals and which do not, and compare EssenceBench's preserved subsets against our robust ranking.