A model that scores well on an expert science benchmark has either learned to apply the underlying theorems or has seen the benchmark during training. Accuracy alone cannot say which. This is a textbook case of what statisticians call non-identifiability: two different explanations of a system fit the observed data equally well, so the data cannot pick between them. Systems biology has met this problem for decades. Its answer was not a better curve fit but a better experiment, one that pushes the system in directions where the competing explanations predict different outcomes. We propose to carry that idea over to contamination auditing. Each expert benchmark problem gets rewritten into a designed family of variants, and the family is built so that "seen this problem" and "can apply this theorem" predict measurably different accuracy patterns. A small latent model is then fit, and it reports how much of a model's score is memorization, how much is skill, and when the data are too weak to tell.
This matters now because the usual tools have stopped working. Detectors that compare a model's loss on seen and unseen text fall to near chance once reasoning models are trained with reinforcement learning or contaminated with worked solutions (On The Fragility of Benchmark Contamination Detection in Reasoning Models). They also require token probabilities that closed models do not expose. A black-box, design-based audit that knows when to abstain would let researchers reread existing expert leaderboards, such as TheoremQA, instead of discarding them.
Core Claim
Contamination and domain skill can be jointly identified from answers alone, using no loss or log-probability signal, provided the evaluation design breaks one specific degeneracy. If perturbing a problem further from its canonical form also makes it harder, then "the model recalls the original and recall fades with distance" and "the model has moderate skill and harder problems defeat it" produce the same falling accuracy curve. The two explanations compensate for each other, exactly as nominal physical parameters and neural weights do in Neural Network Augmented Physics Models for Systems With Partially Unknown Dynamics. The cure is to decorrelate distance from difficulty. The design needs variants that are far from the original yet easy, and variants that are close to the original yet hard. It also needs counterfactual variants where the canonical answer becomes wrong, so that regurgitating it is directly observable.
For model $m$, item $i$ and variant $v$, we model the probability of a correct answer as
$$P(y_{miv}=1)=\pi_{mi}\,g_m(d_v)+(1-\pi_{mi})\,\sigma\big(a_{m,t(i)}-b_{iv}\big),$$
where $\pi_{mi}$ is the probability that model $m$ saw item $i$ (or a close neighbor) in training, $g_m(d)$ is a decreasing recall curve over perturbation distance $d_v$ with $g_m(0)\approx 1$, $a_{m,t}$ is the model's skill on theorem $t$, $b_{iv}$ is the variant's difficulty computed from its executable solution (number of steps and operations), and $\sigma$ is the logistic function. A second observation channel records whether the model outputs the canonical answer on a counterfactual variant where that answer is wrong. This channel loads only on $\pi_{mi}$. The benchmark-level contamination fraction is the average of $\pi_{mi}$ over items.
The identifiability statement is concrete. If $d_v$ and $b_{iv}$ are collinear across the design, the likelihood has a flat ridge along which $\pi$ and $a$ trade off. The Fisher information on $\pi$ then collapses, in the same way that the parameter ridges defeated MCMC in A Bayesian approach for parameter estimation in the extended clock gene circuit of Arabidopsis thaliana. We use that information quantity both to choose the design and to decide when the audit must abstain.
Relation to Prior Work
Two lines of work motivate this proposal by showing the gap.
The first shows that current audits fail. On The Fragility of Benchmark Contamination Detection in Reasoning Models finds that clean reinforcement learning after contaminated fine-tuning shrinks the loss gap between seen and unseen items. It also finds that contamination with worked reasoning chains pushes ten detectors to near chance. That paper diagnoses the failure but offers no replacement. Our audit does not use loss at all.
The second formalizes when auditing is possible. When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits treats contamination as a sparse mixture and shows detectability scales with contamination fraction times separability times the square root of sample size. It assumes a clean reference distribution is available and works with small language models on general text. We borrow its power-calibrated abstention logic. We differ in two ways: our clean reference comes from within-item counterfactual variants rather than an external matched set, and our separability is engineered by design rather than taken as given. Its finding that answer-only exposure is indistinguishable from training drift constrains us, and we test that regime explicitly.
Two works supply the method. Predictive power of non-identifiable models shows that a well-chosen stimulation protocol plus posterior analysis yields reliable predictions under interventions even when individual parameters remain loose. It also shows that forcing identifiability by collapsing the model gives confident but wrong predictions. We take both lessons. We keep the two-component model rather than reducing it to one "robustness score", and we judge it by whether it predicts held-out behavior. Is this Model for Real? supplies a check we reuse: before trusting fits on real data, fit the model to data simulated from itself and compare the parameter landscapes.
On the benchmark side, DyVal and Beyond Memorization generate fresh problems to sidestep leakage. DyVal is limited to symbolic toy tasks. Beyond Memorization synthesizes multi-step problems from arXiv papers and observes no post-cutoff decay. Neither can say how much of a score on an existing benchmark was memorized, and neither rules out that contaminated reasoning generalizes to near neighbors, which is exactly what the fragility paper reports. TheoremQA supplies our domain setting: 800 expert-written questions over 350 theorems with numeric answers, where failure stems from not knowing how to apply theorems. Thinking in a Crowd shows that reasoning models build long justifications for false premises. That finding motivates our counterfactual channel: a model that recalls an answer should rationalize it even after the premise has changed.
Research Program
The first component is a variant compiler for a TheoremQA subset. We restrict it to physics, electrical engineering and finance items whose solutions can be written as short programs. Program-of-Thought prompting already makes this the strongest regime, with GPT-4 at 51.0% in TheoremQA. Each item becomes a parameterized solution program with domain-expert review. From it we generate variants along graded distances: rewording, resampled numeric inputs, counterfactual premises (a changed constant or boundary condition that changes the answer), and sibling problems that apply the same theorem to a new setup. Difficulty is computed from the program, not from model accuracy, so it is not circular. Within each distance level we deliberately include both simplified and lengthened variants, which is how the design decorrelates distance from difficulty.
The second component is the estimator. It is a hierarchical Bayesian fit of the model above, with recall curves and skills pooled across items. It uses profile-likelihood and Fisher-information diagnostics to flag items or models where contamination is not practically identifiable. A design optimizer then selects, under a fixed query budget, which variants to generate so as to maximize information on $\pi$.
The third component is a validation testbed with known ground truth, described under the experiments.
Baselines
The naive reference point is raw accuracy on the original items, which implicitly assumes no contamination. A slightly stronger version reports the drop from canonical to perturbed accuracy. These show what an evaluator currently concludes.
The strongest existing detectors are LiRA and Min-K%-style loss detectors, run exactly as in On The Fragility of Benchmark Contamination Detection in Reasoning Models. They are the right competitor because they are the most reliable known signal and they have white-box access, which we deliberately forgo. That paper reports LiRA at 89.1% AUROC on fine-tuning contamination before reinforcement learning, a drop of roughly 14% after one epoch of GRPO, and 50–58% AUROC across all ten detectors for reasoning-chain contamination of DeepSeek-R1-Distill-Llama-8B. We will run them on the same contaminated checkpoints we audit, with their reference models trained on the same clean data. Beating them means item-level AUROC clearly above that 50–58% band in the reasoning-chain regime, with confidence intervals over seeds excluding it. In the regime where LiRA succeeds, we must stay within its range while using only answers.
Fresh-problem evaluation in the style of Beyond Memorization serves as the reference for skill. Here, newly written sibling problems are scored directly. It is strong because it needs no model. We compare by asking whether our skill estimate, fit only on the variant family, predicts accuracy on a held-out set of sibling problems written later by experts better than this fresh-subset score does at an equal query count.
Three ablations isolate the key ingredient. First, we fit the same estimator on a collinear design where distance and difficulty rise together, at an identical query budget. Second, we drop the counterfactual canonical-answer channel. Third, we replace the two-component model with a single robustness slope. If the decorrelated design gains nothing over the collinear one, the identifiability argument is wrong.
Why not just…?
Why not just report the accuracy drop under perturbed numbers? This is the most serious alternative. For verbatim contamination with difficulty-matched perturbations it may work almost as well. Its weakness is that it is a single statistic answering a two-parameter question. The drop confounds any difficulty shift with forgetting. Under reasoning-chain contamination, which On The Fragility of Benchmark Contamination Detection in Reasoning Models shows generalizes to similar items, the near-variant drop can shrink to zero while scores remain inflated by 11.7%. Our single-slope ablation tests this directly. If it matches the full model on contamination-fraction error across all injection types, we will report that the simpler method suffices.
Why not just evaluate on fresh or post-cutoff problems? Fresh problems such as those in LiveBench or synthesized arXiv questions measure capability going forward. They cannot audit scores already reported on existing expert benchmarks, and expert-grade fresh problems are expensive to write. They are also not immune to leakage: if contaminated reasoning patterns transfer to distributionally similar new items, fresh-but-similar problems inherit the inflation. The held-out sibling experiment compares both approaches on predictive accuracy at equal cost.
Why not just run existing membership detectors? They require token probabilities, which proprietary models withhold. On reasoning models they reach only 50–58% AUROC in the setting that matters most. We include them as a baseline rather than dismiss them.
Why not just reduce the model to one identifiable "contamination score"? Predictive power of non-identifiable models found that collapsing a mechanistic model gave narrow, confidently wrong predictions under intervention. Our single-slope ablation checks whether the same happens here.
Proposed Experiments
The decisive experiment is controlled injection with known ground truth. Following the exposure hierarchy of When Is Benchmark Contamination Detectable? and the two stages of the fragility study, we fine-tune Qwen2.5-7B-Instruct and DeepSeek-R1-Distill-Llama-8B on a random known subset of TheoremQA items. Exposure comes in four forms: verbatim, paraphrased, answer-only, and full reasoning chains. Contamination fractions range from 5% to 50%, across five seeds. Some checkpoints then receive clean GRPO training. On each checkpoint we report three quantities: the error of the estimated contamination fraction, item-level AUROC for recovering which items were injected, and the calibration of posterior intervals. We expect answer-only exposure to be the hardest case. If the audit correctly abstains there rather than reporting a confident zero, that counts as success, not failure.
The identifiability experiment runs the decorrelated and collinear designs at matched budgets. We expect the collinear design to show a flat likelihood ridge, with wide or biased posteriors on the contamination fraction, and the decorrelated design to remove it. We will also simulate response data from the fitted model itself, following Is this Model for Real?, and check that fits on the real checkpoints lie within the scatter of fits on these simulations.
The predictive experiment asks whether estimated theorem skill predicts accuracy on expert-written sibling problems produced after the variant family is frozen. It is compared against raw original-item accuracy and fresh-subset accuracy.
Finally, we apply the calibrated audit to several open models on the original TheoremQA items and report estimated inflation with abstentions. No reported result was found in the file for any per-model contamination estimate on TheoremQA, so these numbers will be new rather than comparable.
The proposal fails cleanly if, on reasoning-chain-contaminated checkpoints, item-level AUROC does not exceed the 50–58% detector band, or if the decorrelated design does not beat the collinear one.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- TheoremQA: A Theorem-driven Question Answering Dataset
- Neural Network Augmented Physics Models for Systems with Partially Unknown Dynamics: Application to Slider-Crank Mechanism
- A Bayesian approach for parameter estimation in the extended clock gene circuit of Arabidopsis thaliana
- Predictive power of non-identifiable models
- Is this Model for Real? Simulating Data to Reveal the Proximity of a Model to Reality
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark