Difficulty Residuals: Catching Benchmark Leakage by Asking What the Question Text Predicts

When a language model scores well on a benchmark, we usually cannot tell whether it reasoned its way to the answers or had already seen the questions during training. The standard detectors look for signs of memorization in the model's token probabilities. These fail in two common situations. Commercial systems do not expose those probabilities. And reasoning models trained with reinforcement learning lose the probability signal: On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that such training drives detector accuracy toward coin-flipping. This proposal uses a different signal that needs only right-or-wrong answers. A regressor predicts how hard each question should be from its wording alone. It is trained only on questions that could not have leaked. We then test whether a model, or the whole population of models, succeeds on some questions far more often than their text says it should. A leaked question looks like a hard question that everyone suddenly finds easy.

The central point is where the difficulty expectation comes from. Existing performance-based detectors take it from other models. ConStat fits a difficulty-correction curve on a fleet of reference models. The adaptive-testing view in Efficiently Measuring the Cognitive Ability of LLMs estimates item parameters from observed model responses. Both inherit whatever contamination the reference models share, and the web-scale corpora that feed every model family make shared contamination likely. A difficulty estimate computed from the question text and fit only on clean items does not have this problem. It gives an outside yardstick that stays valid even when the entire model generation has seen the benchmark.

Core Claim

We claim that a text-derived difficulty prior, calibrated only on contamination-free items, supports two contamination tests that need only black-box correctness. The first is an item-level test. It flags benchmark questions whose empirical difficulty across a model population sits far below what their text predicts. This catches leakage shared by many models, which fleet-calibrated methods cannot see by construction. The second is a model-level test. It checks whether a single target model's pattern of successes fits a standard psychometric model once item difficulties are fixed by the text prior. Because both tests use only whether answers are correct, they should survive the reinforcement-learning "cloaking" that defeats probability-based detectors.

Item Response Theory (IRT) is a psychometric model in which a test-taker with ability $\theta$ answers an item of difficulty $\beta$ correctly with probability

$$P(y=1 \mid \theta, \beta) = \sigma(\theta - \beta),$$

where $\sigma$ is the logistic function. Let $\hat\beta_i$ be the text-predicted difficulty of item $i$ and $\beta_i^{emp}$ its difficulty fitted from a model population's responses. The item-level statistic is the standardized residual

$$r_i = \frac{\hat\beta_i - \beta_i^{emp}}{s_i},$$

where $s_i$ is the predictor's held-out error scale for items like $i$. Large positive $r_i$ means the item is much easier in practice than its wording implies. The model-level statistic is a person-fit score. It is the log-likelihood of target model $m$'s response vector under the fitted $\theta_m$ and the fixed $\hat\beta_i$, standardized against its null distribution. Partial leakage leaves a telltale pattern: the model solves predicted-hard items while failing predicted-easy ones.

Relation to Prior Work

The method's machinery comes from amortized difficulty estimation. Reliable and Efficient Amortized Model-based Evaluation fits a linear model on frozen Llama-3.1-8B embeddings to predict IRT difficulty across 22 HELM benchmarks and 183 models. It reports that this nearly matches empirical calibration and halves adaptive-testing cost. We reuse that predictor design, but we reverse its purpose. Truong et al. treat disagreement between text and response data as error to be minimized. We treat it as the measurement. They also note, without testing it, that leaked items should break the text-difficulty relationship. Our proposal tests exactly this. It also addresses a flaw their pipeline leaves open: their regression targets come from possibly contaminated models. So we calibrate only on items that could not have leaked.

Two works motivate the proposal by exposing the gap. ConStat frames contamination as a failure to generalize from original to rephrased benchmarks, corrected for difficulty by reference models. It detects 98.4% of sample-specific contamination in controlled tests, against 71% for the best prior baseline. We keep its black-box, performance-based logic but replace the reference fleet with a text prior. That replacement lets us detect leakage shared by the fleet, which ConStat cannot, and score new items in a cold-start setting. Efficiently Measuring the Cognitive Ability of LLMs argues that contaminated models produce anomalous IRT response patterns and that the guessing parameter absorbs leakage. It never builds or validates a person-fit detector. We do.

Two other works constrain the design. On The Fragility of Benchmark Contamination Detection shows that clipping in reinforcement-learning objectives compresses the log-probability gap. Chain-of-thought contamination then teaches heuristics rather than strings, so any detector that depends on token probabilities is ruled out. PaCoST compares the model's confidence on original versus paraphrased questions with a paired t-test. It needs white-box access and works only at the benchmark level, and its paraphrase shifts are not adjusted for how hard each item is.

One result cuts against the proposal. On UPSC2M, text embeddings reduced difficulty RMSE by only 7.1% over a dummy regressor. A weak prior yields a weak detector. We address this head-on. Detection power depends on the predictor's held-out correlation with true difficulty, so we measure that correlation per domain and report where the method is and is not usable. We do not assume the strong fit reported on HELM transfers to every setting.

Research Program

The work has three parts.

Building a clean difficulty predictor. Training targets come only from items that the calibration models cannot have seen. We use three sources:

  • rephrased and synthetic counterparts of GSM8k, ARC and MMLU items, in the style ConStat constructs;
  • procedurally generated items from DyVal, whose graph-based generator produces fresh math and logic problems;
  • benchmark items released after the calibration models' training cutoffs.

We fit Rasch difficulties on these clean items from a population of open models. We then train the text regressor on frozen embeddings, following Truong et al. A key risk is that the embedding model has itself memorized the original benchmarks. It might then encode "familiarity" in its embeddings rather than difficulty. We control for this by swapping encoders. We compare an encoder whose training predates the benchmark with a contemporary one, and we check that the residuals do not depend on which encoder we use.

Controlled contamination. We follow the Fragility paper's two-stage protocol with ground truth known. We fine-tune open models on a random subset of AIME, AMC, OlympiadBench and Minerva Math items and their chain-of-thought traces. We then apply GRPO. We run the same design on GSM8k, ARC and MMLU with small Llama and Qwen models. We vary two factors. One is the leaked fraction, from a few percent to the whole benchmark. The other is whether the leakage is confined to the target model or shared across the whole calibration population. The shared case simulates contamination of an entire model generation.

Audit of public models. We apply the item-level residual to the HELM response data and the Open LLM Leaderboard population. We then check the flagged items against known web availability and against n-gram overlap wherever corpora are open.

When the whole benchmark leaks into one model, person-fit alone cannot help, because the model simply looks uniformly strong. For that case we add a third test, a text-matched fresh-item test. We compare the model's fitted ability on the original items with its ability on fresh items, with both sets placed on a common scale by $\hat\beta$. This is ConStat's comparison, with the reference fleet replaced by the text prior.

Baselines

Naive reference points. The do-nothing baseline is raw accuracy with no contamination judgment. For detection AUROC, the naive point is 0.5. We also include item-level scoring by empirical difficulty alone, which flags items that most models solve. This checks whether our residual adds anything beyond "easy items look leaked."

Probability-based detectors. We run LiRA, Min-K% Prob and loss thresholds on identical contaminated checkpoints, with the same held-out member and non-member split. These are the incumbents. The Fragility paper reports LiRA at 89.1% AUROC after supervised contamination. That falls by about 14% after one GRPO epoch. Under post-training chain-of-thought contamination, probability detectors fall to 50–58% AUROC. We count it as beating them if our method holds AUROC well above 58% in that stage-II regime, with confidence intervals from repeated contamination seeds.

ConStat. This is the strongest performance-based competitor. We run it with its reference-fleet difficulty correction, using the same open models that calibrate our predictor and the same rephrased benchmarks. Both methods therefore see identical data and identical inference budgets. In the setting where only the target model is contaminated, matching ConStat's reported 98.4% sample-specific detection would be enough, since our advantage there is the removal of the reference fleet rather than accuracy. In the setting where the fleet shares the contamination, beating it means detecting leakage that ConStat's shifted curve absorbs.

PaCoST. We run it with white-box access on the open models, at the benchmark level, for comparison.

Empirical-β person-fit. This is the key ablation, and the one that isolates the text prior. It is the same person-fit and residual machinery, but item difficulties come from the contaminated model population rather than from text. The whole proposal rests on text beating this ablation in the shared-contamination condition. If it does not, the text prior contributes nothing but cold-start convenience.

Readability prior. A second ablation replaces the embedding regressor with Flesch-Kincaid-style surface features. This tests whether semantic embeddings matter.

For all methods, tuning budgets are matched and hyperparameters are chosen on a separate contamination seed.

Why not just…?

Why not just use difficulties estimated from a reference fleet, as ConStat does? This is the real threat. When only the target model is contaminated, empirical difficulties are less noisy than text predictions, and they may win outright. We expect exactly that and will report it. The text prior's claim rests on the shared-contamination condition. If every calibration model has seen an item, that item's fitted difficulty drops for everyone, so the item shows no anomaly against a fleet-based yardstick. The empirical-β ablation run under shared leakage settles this directly. If text priors do not outperform there, the method reduces to a cold-start tool, and we would say so.

Why not just use token-probability detectors like Min-K% or LiRA? They require white-box access. The Fragility paper also shows they drop to 50–58% AUROC once reasoning models are trained with reinforcement learning on contaminated chain-of-thought data. Our stage-II replication tests whether a correctness-only signal survives where theirs does not.

Why not just evaluate on fresh or post-cutoff questions and compare scores? A score drop on fresh items means contamination only if the fresh items are equally hard. Unveiling Causal Reasoning in Large Language Models and DyVal observe such drops but cannot separate leakage from a harder item set. Matching difficulty is precisely what $\hat\beta$ supplies. So this alternative is a component of our method, and the whole-benchmark leakage condition tests it.

Why not just use readability scores as the difficulty prior? Surface features track vocabulary rather than reasoning depth. That is a poor fit for math and logic items, whose difficulty lies in the reasoning they require. The readability ablation measures how much of the detector's power disappears without embeddings.

Why not just match n-grams against training corpora? Rephrased leakage bypasses it, as ConStat documents. It also needs corpus access that closed models do not provide. We use it only as partial ground truth in the public audit.

Decisive Checks

The proposal succeeds only if three things hold on the controlled stage-II reasoning contamination.

First, the text-prior tests must detect leaked items at AUROC clearly above the 50–58% range reported for probability detectors. Second, under shared fleet contamination, the text-prior residual must detect leakage that both ConStat and empirical-β person-fit miss. Third, detection power must track the predictor's held-out difficulty correlation in the way the logistic model implies. Weak predictors, like the one reported on UPSC2M, should yield proportionally weak detection rather than spurious flags.

Two further results would kill the idea. One is residuals that shift substantially when the embedding encoder is swapped, which would mean the encoder is reading familiarity rather than difficulty. The other is clean-item residuals that are as extreme as leaked-item residuals.