Self-Ranked Difficulty as a Contamination Probe for Reasoning Models

Large language models are often scored on test questions they saw during training. The usual detectors look at how confident the model is on each question, and they have stopped working on reasoning models. A short phase of reinforcement learning, which is the standard last training step for these models, erases the confidence gap between seen and unseen questions while the inflated scores stay. We propose to stop asking the model how sure it is about one question and instead ask it to order questions by difficulty. Each original benchmark question is mixed into a pool with a rewritten twin that needs the same reasoning but has different surface details. The model then sorts the whole pool through pairwise "which of these two is harder?" judgments. A model that reasons from scratch should place a question and its twin close together. A model that has memorized the original should push it toward the "easy" end, or solve it while still ranking it as hard. The distance between twins in the model's own ordering is the contamination signal.

Sorting a pool of $n$ items with pairwise judgments naively takes $n^2$ model calls. This is where work on fast ranking matters. Pairwise ranking prompting showed that pairwise judgments from a language model can be fed into a sorting algorithm and cost $O(n \log n)$ calls, with an order-swapping trick that removes position bias (Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting). The resulting statistics can then be computed by sorting rather than by enumerating pairs. The proposal reuses that machinery and points it at a different question: whether a model's own ordering of questions gives it away.

Core Claim

We claim that rank displacement between an original item and its twin, measured inside a model's self-sorted difficulty ordering, detects item-level contamination in reasoning models after reinforcement learning. We expect it to do so clearly better than log-probability detectors. We also expect it to add information beyond simply checking whether the model solves the original but not the twin, at the same budget of model calls. The test needs no clean reference models, and it is calibrated against the model's own behavior on fresh questions.

The motivation comes from On The Fragility of Benchmark Contamination Detection in Reasoning Models. That paper shows two things. Clean GRPO or RAFT++ training after contaminated supervised fine-tuning collapses Loss, Min-K%, LiRA and seven other detectors toward chance. And when contamination takes the form of memorized chain-of-thought (written-out reasoning steps), it carries over to unseen similar questions, so confidence gaps vanish. The paper explicitly leaves open detectors that do not rely on log-probability gaps. It also does not test whether relative orderings over difficulty-matched pools survive. Our probe is behavioral and relative. It asks for a comparison rather than a probability, and the twin controls for topic and difficulty.

A second constraint comes from LiveBench. It reports that models are poor judges of problems they cannot solve, so a model's sense of difficulty is tied to its own ability. We therefore never compare the model's ordering to an external difficulty scale. We only compare twins against each other, and we calibrate on fresh questions where we know the model cannot be contaminated.

Research Program

For a benchmark of $N$ items, we build a twin $i'$ for each item $i$. For math, the twin changes numbers and entities and is re-verified by script. For multiple choice, it is a rewritten stem with shuffled options, filtered so the answer is unchanged. The model under test sorts the pooled $2N$ items with PRP-Sorting (heapsort). Each comparison is asked in both presentation orders, and conflicts are treated as ties, following the pairwise ranking prompting paper. We also record whether the model solves each item, using several samples.

Let $r(x)$ be the position of item $x$ in the model's ordering, from 1 (easiest) to $2N$. Let $y_x \in \{0,1\}$ record whether the model solved $x$ by majority over samples. The per-item score combines displacement and solve asymmetry:

$$ s_i = \frac{r(i') - r(i)}{2N} + \gamma \,(y_i - y_{i'}), $$

Here $\gamma \ge 0$ is a single weight fit on the calibration split. A large positive $s_i$ means the original looks easier to the model than its twin, or is solved when the twin is not.

Calibration uses the same model on fresh items: LiveBench questions released after the model's training cutoff, together with their own twins. This gives the null distribution of $s_i$ for that specific model. That null replaces ConStat's population of reference models, whose own contamination shifts the expected curve (ConStat). For a benchmark-level verdict, we compare the original and twin ranks with a Wilcoxon–Mann–Whitney-type statistic. Following the sorting identity used to make such statistics linear after sorting (A fast algorithm for learning large scale preference relations; Luo et al.), this costs nothing beyond the sort.

At $N=500$, the full sort with order swapping takes roughly $2 \cdot 2N\log_2 2N \approx 2\times10^4$ short comparison prompts. We also test PRP-Sliding-$K$ as a cheaper partial sort that only finds the extremes of the ordering.

Baselines

The naive reference point is chance at item level (50% AUROC, the area under the curve that trades off true and false detections). We also report raw accuracy on the original items, which is what a leaderboard reports when no one audits it.

The log-probability family consists of Loss, Min-K% and LiRA, run exactly as in the fragility paper. They tell us whether our replication of the collapse is faithful. The fragility paper reports LiRA at 89.1% AUROC before reinforcement learning, a drop of about 14 points after one epoch of clean GRPO, and all ten methods at 50–58% AUROC under chain-of-thought contamination. PaCoST compares the model's pointwise P(True) on original versus rephrased questions with a paired t-test (PaCoST). It is the closest confidence-based relative of our probe. We run it on the same twins.

The strongest competitor is the item-level solve gap. This is a ConStat-style performance discrepancy restricted to the same twins: the original is solved and the twin is failed, estimated with as many samples as our method's total call budget allows. ConStat reports detecting 98.4% of sample-specific contamination in controlled experiments, against 71% for the best earlier baseline (ConStat). Performance inflation is also exactly what survives reinforcement learning in the fragility paper. Fairness here means identical twins, identical models, and matched total generated tokens, with the solve-gap baseline allowed to spend our entire sorting budget on extra samples. We beat it only if our score raises item-level AUROC by a margin outside the bootstrap interval at matched compute. The margin has to hold in the chain-of-thought contamination stage, where the solve gap is expected to shrink because contamination carries over to twins.

A psychometric baseline fits item response theory to all models' responses. Item response theory models each question's difficulty and how sharply it separates strong from weak models, and its person-fit residuals flag hard items solved by weak models, as proposed in Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective. It is fit on the same pool of contaminated and clean model checkpoints.

Three ablations isolate the ranking ingredient. The first replaces sorting with a pointwise 1–10 difficulty rating. The second asks only the direct question "is $i$ or $i'$ harder?" once per pair, an $O(N)$ cost with no global ruler. The third removes the order swapping.

Why not just…?

Why not just measure the solve gap between originals and twins? This is the serious threat. It may well be sufficient after supervised-fine-tuning contamination plus reinforcement learning, since the fragility paper shows the 8.8% inflation remains. Our bet is narrower. Under chain-of-thought contamination, solving carries over to twins, so the solve gap loses power, while a familiarity bias in difficulty judgments might not. The matched-compute comparison in the chain-of-thought stage settles this. If the solve gap matches us there, we will report the displacement term as unnecessary and recommend the cheaper test.

Why not just ask the model to rate difficulty, or use PaCoST's P(True)? Pointwise ratings by language models are poorly calibrated, which is precisely why pairwise prompting beat pointwise and listwise ranking in the pairwise ranking prompting paper. PaCoST also reads a confidence signal of the kind reinforcement learning flattens. The pointwise-rating ablation and the PaCoST baseline test this directly on the same twins.

Why not just use item response theory person-fit statistics? Person-fit needs many models answering the same items. That brings back the reference-population problem: if most peers are contaminated, contaminated items look normal. The fragility results also suggest reasoning-model training shifts item difficulty itself. Our score is computed from one model plus fresh calibration items. We will report how both methods degrade as the fraction of contaminated peer checkpoints grows.

Why not just evaluate on fresh benchmarks like LiveBench? Fresh benchmarks avoid contamination going forward. They cannot audit scores already reported on static benchmarks, and they are restricted to tasks with script-checkable answers. We use LiveBench as the clean calibration source rather than competing with it.

Proposed Experiments

The main testbed replicates both stages of the fragility paper on AIME, AMC, GPQA Diamond, OlympiadBench and Minerva Math. A random half of the items is injected into supervised fine-tuning, either as answers in Stage I, followed by clean GRPO, or as full reasoning traces in Stage II with DeepSeek-R1-Distill-Llama-8B. This gives item-level ground truth. The primary metric is item-level AUROC at matched model calls, tracked across GRPO epochs. Secondary metrics are benchmark-level power at a 5% false-positive rate on uncontaminated control checkpoints, and calls needed to reach a target AUROC.

A second experiment uses GSM8K and ARC with controlled contamination of open models. As an external sanity check, we test whether the method flags the cases ConStat reports, such as $\delta \approx 8.3\%$ for Mistral-7b-v0.1 on GSM8K, without any reference models.

Three decisive checks follow. First, displacement AUROC must stay well above the 50–58% range of log-probability detectors after reinforcement learning. Second, it must beat the matched-compute solve gap in Stage II. Third, the full sort must beat the direct pair-only ablation. The third check is what justifies the $O(N \log N)$ sort over a single $O(N)$ comparison per pair. If it fails, the twin comparison alone carries the signal, and we would say so.