Prerequisite Audits: Detecting Cloaked Benchmark Contamination by Probing the Causal Structure of Competence

When a language model has seen a benchmark during training, its score on that benchmark goes up for reasons other than skill. The standard way to catch this is statistical. A membership-inference detector checks whether the model assigns unusually high likelihood to the test items. That signal can now be erased. A recent audit shows that one epoch of clean reinforcement learning after contaminated fine-tuning drives these detectors toward chance, and that contaminating an already-trained reasoning model with worked solutions inflates pass@1 by 11.7% while ten detectors sit at 50–58% AUROC (On The Fragility of Benchmark Contamination Detection in Reasoning Models). Leaderboards therefore have no reliable way to tell earned scores from leaked ones.

We propose to stop asking whether the model has seen the text, and to ask instead whether its success on each problem is causally supported by success on that problem's prerequisites. Every multi-step problem has a prerequisite graph: the intermediate quantities or facts that must be established before the answer follows. The causal curriculum literature in reinforcement learning treats such graphs as discoverable objects that an agent intervenes on to test its competence. Genuine competence leaves a structural fingerprint. If a model can solve the whole problem, it can usually solve the parts, and when an upstream quantity is changed, its downstream answers change accordingly. Our hypothesis is that contamination, including contamination hidden by reinforcement learning, produces success that is disconnected from its prerequisites. That disconnection can be measured with targeted sub-problem queries and interventions. A likelihood signal is never needed, so it does not matter that one has been erased.

Core Claim

The claim is that item-level "prerequisite inversion" is a contamination signal that survives reinforcement-learning cloaking where likelihood-based detectors fail. Inversion has two parts: the model answers the final question correctly while failing parent sub-questions, or its answers fail to track interventions on parent values.

Stated as a conditional-independence test, the claim reads as follows. Let $Y_i$ be correctness on item $i$'s final answer and $P_i$ the vector of correctness on its prerequisite sub-queries. For items the model genuinely solves, $Y_i$ should depend strongly on $P_i$. For memorized items, $Y_i$ should be close to independent of $P_i$, because the answer is retrieved rather than derived.

The idea borrows from three places. It reuses the spurious-link pruning idea of D3HRL, which uses conditional mutual information tests to separate causal from coincidental transitions in grid worlds. Here the test separates derived answers from retrieved ones. It reuses the intervention-ranking idea of HRC, which prioritizes subgoals by estimated causal effect rather than exploring uniformly. Here the ranking decides which sub-queries to spend a limited probe budget on. Finally, it turns the observation from Compositional Causal Reasoning Evaluation in Language Models into a contamination test. That paper found that GPT-4o often gets global causal answers right while failing intermediate ones, a pattern it calls "valid-inconsistent." The paper treats this as a capability property and never connects it to contamination; we test whether contamination produces it.

The claim can fail. The fragility paper argues that CoT-contaminated models "internalize the underlying reasoning process." If that is literally true, contaminated items will show normal prerequisite dependence and our audit will find nothing. That outcome would itself settle the open tension between the fragility paper and the compositional-reasoning results on valid-inconsistent behavior.

Research Program

The work has three components.

Probe construction. For each benchmark item we construct a code-verified prerequisite graph. Each node is a sub-question whose answer can be computed by executing code.

  • On fully synthetic tasks we use the graph generator of DyVal for arithmetic, Boolean logic and reachability. The graph is exact and node answers are free.
  • On real benchmarks we follow DARG. An LLM extracts the reasoning graph from GSM8K items and from Minerva Math and OlympiadBench items, and a code-execution check verifies it. Items whose graphs fail verification are discarded.

We report the extraction yield honestly, because DARG-style generators simplify or hallucinate on complex problems. We expect low yield on AIME-level items and do not plan to rely on them.

Two probe types per node. A prerequisite query asks for the node's value directly, posed as a standalone question with the downstream context removed. An intervention query changes the value of one parent node in the problem text and asks for the final answer. Its ground truth is recomputed by code, which serves as a textual analogue of $\text{do}(X)$.

Adaptive probe selection. Probing every node of every item is expensive. In the spirit of HRC's causal-effect ranking, we select up to $k$ probes per item greedily, choosing those with the highest expected information gain about the item's membership label under a small Bayesian model of how genuine and memorized competence generate probe outcomes. The item-level audit score is the posterior log-odds of "unsupported success," computed only on items the model answers correctly. Aggregating gives a model-level estimate of inflation: raw accuracy minus prerequisite-supported accuracy.

Baselines

The naive reference point is raw benchmark accuracy with no audit. Against it, our model-level inflation estimate must recover a nonzero gap whose size tracks the true contamination dose.

The strongest existing competitors are the likelihood-based detectors evaluated in the fragility paper, chiefly LiRA and loss thresholding. The comparison will be exact. We re-implement that paper's pipeline: contaminated fine-tuning on a member half of each benchmark, then one epoch of clean GRPO, with the other half as non-members. Both LiRA and our audit are scored on the same member/non-member split with item-level AUROC. LiRA gets the same reference-model budget the fragility paper used.

The reported numbers come from On The Fragility of Benchmark Contamination Detection in Reasoning Models:

  • LiRA reaches 89.1% AUROC before reinforcement learning.
  • After clean GRPO, LiRA drops by about 14% and loss by about 16%, with reference-free detectors near 50%.
  • In the post-LRM setting with DeepSeek-R1-Distill-Llama-8B, all detectors fall to 50–58% AUROC.

Beating this baseline means exceeding 58% AUROC in the Stage II setting, and exceeding post-GRPO LiRA in Stage I, by a margin larger than the seed-to-seed spread across three contamination runs.

The second competitor is a fresh-instance gap. We compare accuracy on member items with accuracy on DyVal- or DARG-style regenerated isomorphs of the same items, at matched graph depth. This baseline tests whether our structure-sensitive probes add anything over simply re-sampling the problem.

The third competitor is the consistency evaluation of the compositional-reasoning paper applied without our probe selection. It queries all graph cut points and scores the item valid-inconsistent if any intermediate fails.

Two ablations isolate our key ingredients. The first replaces adaptive selection with $k$ random nodes at equal query budget. The second drops intervention queries and keeps only prerequisite queries. Each must cost measurable AUROC for that ingredient to earn its place.

Why not just…?

Why not just run a better membership-inference detector? The fragility paper gives both theory and experiments showing that PPO/GRPO importance sampling and clipping compress the likelihood gap that every such detector relies on. A cleverer threshold on a vanished signal is unlikely to help. Our Stage I and Stage II comparisons report LiRA side by side with the audit on identical models, so the claim is tested rather than assumed.

Why not just measure accuracy on freshly generated isomorphic problems? This is the most serious threat. The fragility paper reports that CoT-contaminated models carry their boost to unseen, distributionally similar splits. If so, freshly generated isomorphs would be inflated too, and the fresh-instance gap would understate contamination. Our probes deliberately break the template: they ask for intermediate values in isolation and change parent values. Whether that matters is an empirical question. If the fresh-instance gap tracks true inflation as well as our audit does, we will report that the cheaper method suffices and that prerequisite structure adds nothing.

Why not just use the existing compositional consistency test? That test, from Compositional Causal Reasoning Evaluation in Language Models, probes every cut point on synthetic causal models with no contamination ground truth. We supply three things it lacks: controlled contamination with known members, graphs extracted from real benchmark items, and intervention queries. The full-probe baseline shows directly whether adaptive selection saves queries without losing detection power.

Why not just paraphrase the questions? Paraphrasing in the style of Meta Probing Agents cut GPT-4-Turbo's MMLU accuracy by 15.7%. However, it needs a strong judge model to certify equivalence, and it leaves the problem's structure untouched. A²RBench found that models leaned on token patterns, losing 17.7% when basic symbols were remapped, which suggests surface variation and structural reliance are different axes. We include a paraphrase control at matched budget to test whether it explains our signal.

Proposed Experiments

Tier A: fully controlled synthetic setting. We fine-tune a base model on CoT solutions for a fixed member set of DyVal instances at depths D2–D4, then run clean GRPO. Graphs are exact, so prerequisite and intervention ground truth is perfect, and contamination dose (repetitions of member items) can be varied.

Tier B: real benchmarks. We reproduce the fragility paper's Stage I pipeline on GSM8K, Minerva Math and OlympiadBench subsets with verified graphs, and its Stage II setting on DeepSeek-R1-Distill-Llama-8B. A clean twin model, trained identically without contamination, gives the true per-item inflation.

Metrics. We report item-level AUROC for membership and the correlation between estimated and true model-level inflation across doses. We also report the conditional mutual information between final correctness and prerequisite correctness, computed separately for members and non-members, which is the most direct test of the core claim.

Decisive checks. Three results matter most.

  • In Stage II, where likelihood detectors sit at 50–58%, the audit must separate members from non-members.
  • The audit's inflation estimate must rise monotonically with contamination dose, while the clean twin shows near-zero inflation. This guards against the audit merely flagging hard items.
  • The audit's advantage must persist after GRPO, the condition under which the baselines collapse.

A null result on the conditional-independence test would mean that contaminated CoT really does produce structurally supported competence. We would publish that as a finding about what contamination teaches.