Contamination Leaves a Verification Fingerprint

When a language model has seen a test problem and its solution during training, it can often reproduce the answer without having learned how to solve it. Current contamination detectors check whether the model can regurgitate benchmark text, for example by asking it to fill in masked words of a test question. A light rephrasing of the leaked data defeats them. This proposal asks the model to check solutions rather than solve problems.

A model that has memorized one canonical solution should judge other solutions by how far they depart from that memory, not by whether they break a rule. Concretely, it should wrongly reject valid solutions that take a different route, and it should flag "errors" wherever the text diverges from its memory rather than where a constraint is actually violated. This pattern is a property of the model's behavior, not of the training text, so it should survive rephrasing of the leaked data. Because solutions can be generated procedurally, we can contaminate models on purpose and measure the effect directly rather than guessing.

Core Claim

Benchmark contamination changes the gap between solving a problem and checking a solution, and the change is specific enough to detect. On a contaminated item, a model accepts the canonical solution, rejects correct non-canonical solutions far more often than on clean items of matched difficulty, and places its error reports near points of textual divergence instead of the true broken step.

The gap is measured at three levels of a task hierarchy. At the first level the model retrieves or checks one value. At the second it checks one local step against its rule. At the third it judges a whole solution. The fingerprint should be strongest at the local-step level, where a reasoning model checks the rule and a memorizing model compares against memory.

Why this intersection, and what is already done

Several prior results motivate the claim. The Self-Verification Cliff shows that frontier models generate correct answers far more often than they can pick them out of their own samples, and that the gap widens with capability. It measures this only on final integer answers of 86 static AIME problems, which are themselves exposed to leakage. On the self-verification limitations of LLMs reports a 97.09% false-negative rate in Mystery Blocksworld, an obfuscated version of the classic block-stacking planning puzzle; a false negative here means rejecting a valid plan. It attributes this to approximate retrieval. We turn that observation into a measurable test: false rejection of valid alternatives is exactly what retrieval-based checking predicts.

MR-GSM8K already splits verification into three levels: a correct/wrong judgment, locating the first wrong step, and explaining the error. That split is the hierarchy we borrow. Its 2,999 items are built on the heavily leaked GSM8K and use error traces from a single model, so they cannot separate memorization from skill.

Other work supplies the method or constrains it. DyVal generates problems from directed acyclic graphs (DAGs: networks of computation steps with no loops) at controlled depth. BeyondBench produces more than $10^{15}$ instances per task, with exact solvers providing ground truth. Both evaluate only generation. We reuse their style of generator, but we also produce matched checking tasks: valid alternative solutions and solutions with errors injected at known steps.

On detection, TS-Guessing asks models to fill masked slots in test items. Work on rephrased samples and on detection fragility in reasoning models shows that such text-based signals break under paraphrase. This is the gap our behavioral signal targets.

No paper in the file measures how contamination shifts the solve-versus-check gap, or does so under controlled contamination doses. The nearest works are BeyondBench, which generates fresh problems but tests only solving, and MR-GSM8K, which tests multi-level checking but on fixed, leaked data. Neither treats the shape of checking errors as evidence of memorization.

There is a competing hypothesis that we take seriously. A memorized solution could help a model find injected errors, because the model can compare against memory. If that held, contamination would inflate error-location accuracy and look like skill. Our design separates the two hypotheses by including valid solutions that differ from the canonical one:

  • A model that compares against memory gains on injected errors but loses on valid alternatives.
  • A model that checks the rules gains or loses on both together.

Research Program

Generator. We build one generator covering four domains with exact checkers:

  • DAG arithmetic in the style of DyVal.
  • Graph coloring.
  • 3-SAT near and away from the hard threshold studied in the 3-SAT characterization. 3-SAT asks whether true/false values can satisfy a set of three-variable clauses.
  • Blocksworld planning.

Planning is included on purpose. Prior work (Valmeekam et al.) finds that checking is easier than generating there, while in math-style tasks generation outpaces checking. The fingerprint has to be read against each domain's own clean baseline.

Tasks per instance. For every instance the generator emits a canonical solution and several alternatives:

  • Valid alternatives: a different valid evaluation order, coloring, satisfying assignment, or plan. Each is scored by its edit distance from the canonical solution.
  • Corrupted solutions: a single rule violation injected at a known step and depth, with the surrounding text kept either close to or far from the canonical one.

These yield paired solve and check tasks at all three levels. Checking at the second and third levels is graded against the solver, with no LLM judge.

Controlled contamination. We fine-tune open-weight models on a held-out pool of instances, at the Qwen2.5-1.5B and 7B scale used in Learning to Self-Verify. The doses are:

  • no exposure;
  • problem text only;
  • problem plus canonical solution;
  • the same content paraphrased and re-templated, the case that defeats text detectors;
  • a mix of continued pretraining on general text with sparse leakage.

Known contamination labels let us build a detector from the checking profile: valid-alternative rejection rate, how error-location accuracy varies with distance from the canonical text, and the gap between solving and checking at each level. We then estimate how well this detector separates contaminated from clean items.

Transfer to real leakage. Finally, we apply the probes to frontier and open models on classic instances that circulate widely online (well-known Game of 24 sets, standard Blocksworld suites) versus freshly generated instances matched in difficulty. We report whether the fingerprint appears where leakage is suspected.

Baselines

The naive reference is generation accuracy alone. It asks whether contaminated items are simply solved more often. Under controlled doses this detects memorization trivially in aggregate, but it cannot flag an individual item without a clean twin, and it cannot tell memorization from easier items.

The strongest existing text-based detector is TS-Guessing from Investigating Data Contamination. That paper reports 57% exact match for GPT-4 and 52% for ChatGPT on masked-slot guessing over MMLU, and near 0% for Llama 2-13B. We run it on our generated items, masking non-essential tokens, with the same models and the same fine-tuned checkpoints. We add a per-item likelihood-gap detector, which compares the model's probability for the original versus a perturbed item. We tune its threshold on the same validation split our detector uses. No reported result was found on Lacuna for TS-Guessing or any detector under controlled, paraphrased contamination of procedurally generated reasoning items, so both baselines are measured here.

The strongest generation-side competitor is the dynamic-twin comparison used by DyVal and BeyondBench: compare accuracy on an item against fresh isomorphic variants. BeyondBench reports that models scoring above 90% on GSM8K have their accuracy roughly halved on its uncontaminated variants. We implement a twin-gap score per item, using the same number of model calls as our probes.

The asymmetry-only baseline is the answer-level self-selection gap, pass@8 minus self-best@8, from the Self-Verification Cliff. Pass@8 counts a problem as solved if any of eight samples is correct; self-best@8 scores the sample the model itself picks. That paper reports pass@8 of 54.7% and self-best@8 of 41.9% for GPT-5.4-mini on AIME, and 40.7% and 34.9% for Gemini-3.5-flash.

Two ablations isolate our key ingredient:

  • Removing the valid-alternative probes, which leaves only injected errors.
  • Removing step-level probes, which leaves only whole-solution judgments.

Beating the baselines means a higher area under the ROC curve (AUROC) for item-level contamination detection under the paraphrased dose. AUROC is the probability of ranking a contaminated item above a clean one. The comparison is made at an equal query budget, and our detector should lose little AUROC between verbatim and paraphrased contamination while the text-based detectors drop.

Why not just…?

Why not just use dynamic twins and compare accuracy? This is the most serious competitor, and it may match us on verbatim contamination. Its weakness is that a twin is a different problem: any accuracy gap mixes memorization with differences in difficulty. It also says nothing about how the model fails. Our probes stay on the same instance. The experiment settles this by reporting both detectors at each dose. If twin gaps match our AUROC at every dose, including paraphrase, the fingerprint adds only diagnostic value, and we will say so.

Why not just run TS-Guessing or likelihood detectors? They measure recall of training text. The rephrasing studies cited above show that this signal can be removed while the leaked solution still helps. Our paraphrased-dose condition tests exactly this. If text detectors hold up there, the motivation for a behavioral signal weakens.

Why not just measure the answer-level self-selection gap? It is cheap and already documented, but it pools all failures into one number. It cannot distinguish rejecting a valid alternative from missing a real error, and those are the two errors our competing hypotheses predict in opposite directions. The ablation that drops step-level and valid-alternative probes tests whether the cheaper measure suffices.

Why not just ask the model to self-critique? Prior evidence argues strongly against relying on it. Self-critique dropped Graph Coloring accuracy from 16% to 2%, and MR-GSM8K found that few-shot critique examples biased models toward guessing a verdict rather than checking. We therefore use self-critique as an object of measurement, never as a grader.

Proposed Experiments and Decisive Checks

The first decisive check runs on fine-tuned open models. If contamination works through memory comparison, the rate of rejecting valid alternatives should rise with dose. That rate should grow with the alternative's edit distance from the canonical solution, and it should stay flat on clean items of matched difficulty. If this does not happen, the core claim is false.

The second check is error localization on corrupted solutions:

  • Corruptions far from canonical text should be "found" more easily by contaminated models even when the injected error is minor.
  • Corruptions near canonical text should be missed.
  • Clean models should show no such distance dependence.

The third check is robustness. We compare detector AUROC between the verbatim and paraphrased doses for our profile, TS-Guessing, likelihood gaps, and twin gaps, each at a matched query budget.

The fourth check concerns the hierarchy. We test whether the fingerprint concentrates at the local-step level as predicted, and whether it reverses or vanishes in Blocksworld, where clean checking is already easier than generation.

The last check is the real-leakage transfer to widely circulated classic instances versus fresh matched ones. It is suggestive rather than conclusive, since true contamination labels are unknown there. We report it as such.