When a language model answers a benchmark question correctly, we cannot easily tell whether it reasoned or recalled. A reasoning model that concludes "this protein binds caffeine" should also agree that it binds a stimulant, because caffeine is a kind of stimulant. A model that has simply memorized the answer string has no reason to get the implied parent claim right. This note proposes to use that asymmetry as a behavioral test. For each benchmark item whose answer sits inside a known taxonomy, we automatically generate the questions its answer logically entails higher and lower in the hierarchy. We then measure how often the model gets the item right while contradicting what the item implies. The prediction is that leaked items show more of these contradictions than clean items of matched difficulty.
The timing matters because the standard contamination detectors have stopped working on the models people now care about. These detectors look for signs that a model has seen the exact text, such as unusually low loss or confident token probabilities. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that a single epoch of reinforcement-learning post-training erases the loss gap these detectors rely on. It also shows that contaminating a model with chain-of-thought data drives ten detectors to 50–58% AUROC, roughly chance. A test that asks whether the model's answers hang together logically does not depend on token probabilities, so it could survive where those detectors fail. The same probes also measure a basic reasoning limitation: whether models carry "is-a" inheritance through a hierarchy at all.
Core Claim
Our claim is that a hierarchical coherence gap is a useful contamination signal that stays usable after reinforcement-learning post-training. The gap is the excess rate at which a model answers an item correctly but violates the ancestor or inherited-property claims that the answer entails. By "usable" we mean the signal keeps discriminating leaked from clean items after that training, in the regime where loss-based and verbatim detectors collapse. We also claim that it adds information beyond shallow perturbation consistency.
To make this precise, fix an item $i$ that the model answers correctly. Let $Q_i$ be its set of entailed probes, meaning questions about ancestors of the answer or properties inherited from them. The per-item violation score is
$$s_i = \frac{1}{|Q_i|}\sum_{q \in Q_i} \mathbb{1}[\text{model contradicts } q],$$
and the benchmark-level gap is $\Delta = \bar{s}_{\text{suspect}} - \bar{s}_{\text{matched clean}}$. Here "matched clean" means post-cutoff or synthetic items paired on difficulty and on how typical the entity is of its category.
The idea draws on three lines of prior work. Two findings motivate it. TAXI edits a subject's category in a model and checks whether the implied properties update. In-context editing reaches only 0.47 consistency against a human baseline of 0.86, which shows that models do not store categories as coherent hierarchies. TAXI studies editing, not contamination, and never asks whether incoherence tracks exposure during training. Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones? pairs hard questions with easier sub-problems they contain and finds that GPT-4 Turbo solves the easy one given the hard one 92.2% of the time. The authors explicitly do not control for contamination. We borrow its conditional-consistency form but replace the ladder of difficulty with a ladder of taxonomy, and we make contamination the object of study.
A third line constrains the design. CAP already uses consistency under perturbation as a contamination detector. Its perturbations are shallow: shuffled answer options, changed distractor details, paraphrases. Our probes instead require an inference the original item does not state, and we treat paraphrase consistency as the ablation that must be beaten.
Research Program
The program has three parts.
The first part is a probe generator. For items whose gold answer is a taxonomic entity, the generator produces entailed questions about ancestors, inherited properties, and exclusions (for example, "not a sibling class"). Items come from TAXI's 41 categories and 164 subjects, and from the biology, chemistry and medicine subjects of MMLU. We also build procedurally generated fictional taxonomies in the spirit of BeyondBench. These are invented kinds with invented properties, whose depth we control and which cannot have been seen in training. They give a floor for how much incoherence comes from reasoning failure alone, measured separately at each depth.
The second part is a controlled contamination study, where we know which items leaked. We randomly assign half the probe-bearing items to a "leaked" set. Random assignment balances difficulty and typicality by construction. We then contaminate open models in the two ways the Fragility paper describes: fine-tuning followed by clean reinforcement learning with GRPO (a group-based policy-optimization method), and direct chain-of-thought contamination. Crucially, the probes themselves are never trained on. Only the items and their solutions are.
The third part is a field audit. We compute $\Delta$ on released models for MMLU subsets, using TAXI-style synthetic items as the clean reference. We compare the result against partial-prompt reconstruction, the audit used in Reasoning or Memorization?.
Baselines
The naive reference is raw accuracy with no contamination score, which amounts to guessing at 50% AUROC. It is joined by a matched-difficulty "clean" violation rate, which tells us how incoherent the model is anyway.
The strongest existing detectors are the loss-based and reference-based membership methods evaluated in On The Fragility of Benchmark Contamination Detection in Reasoning Models: Min-K%, LiRA, Loss and Verbatim. That paper reports that under chain-of-thought contamination all ten methods fall to 50–58% AUROC, and that LiRA loses about 14 AUROC points after one GRPO epoch. We run these detectors on exactly the same contaminated checkpoints and item splits. Wherever a method needs a reference model, it gets one with the same compute budget. Beating them means clearly exceeding the 58% ceiling in the chain-of-thought setting, with confidence intervals from repeated random splits.
CAP is the closest behavioral competitor. That paper reports 21 leaking FinEval questions but gives no AUROC; no reported AUROC for CAP was found on Lacuna. We reimplement its three perturbation types with the same number of queries per item as our probes.
Partial-prompt completion is the third comparison. On MATH-500, Qwen2.5 reaches 65.8% exact match from an 80% prompt prefix, per Reasoning or Memorization?. We apply it to the same items.
The key ablation replaces entailed hierarchical probes with paraphrases of the item itself, matched in number and length. If this ablation matches our method, the taxonomic ingredient adds nothing.
Why not just…?
Why not just use CAP-style paraphrase consistency? This is a real threat. A memorizing model may fail paraphrases as readily as it fails entailed questions, in which case hierarchy is decoration. The paraphrase ablation above settles it directly. We commit to reporting the proposal as negative for the hierarchical ingredient if the gap in AUROC between the two conditions is within noise. We expect a difference because paraphrases test recognition of the same fact. Entailed probes test whether the fact is linked to anything else, and that link is exactly what TAXI shows is weak.
Why not just evaluate on procedurally generated benchmarks and skip detection? Dynamic suites such as BeyondBench and LiveBench avoid contamination going forward. They cannot say whether published scores on existing static benchmarks are inflated. That audit question is the one this proposal answers. We do use synthetic taxonomies, but only as the clean reference.
Why not just use partial-prompt reconstruction? Reconstruction detects verbatim memory. The Fragility results indicate that chain-of-thought contamination produces generalized confidence rather than verbatim recall. We test partial-prompt reconstruction on the same checkpoints to see whether it degrades the same way.
Why not just enforce coherence with constrained decoding? Methods such as Seq2Tree and C-HMCNN make hierarchical outputs coherent by construction. These are fixes, not diagnostics. Applying them would mask the very violations we measure. They are relevant only as a later intervention.
Decisive Checks
Four results would decide the question. First, in the controlled study, $s_i$ must separate leaked from clean items with AUROC above 58% after chain-of-thought contamination and after GRPO, where the loss-based baselines are near chance. Second, it must beat the paraphrase ablation. Third, on fictional taxonomies, violation rate should rise with depth, which would confirm that the probes measure inheritance reasoning. The gap attributable to contamination must remain after we subtract this depth-matched floor. Fourth, we test the failure mode we worry about most. If chain-of-thought contamination data happens to spell out ancestor facts, the gap may vanish. We will vary whether the leaked solutions mention ancestors, to map when the signal survives and when it does not.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- TAXI: Evaluating Categorical Knowledge Editing for Language Models
- Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?
- CAP: Data Contamination Detection via Consistency Amplification
- Measuring Massive Multitask Language Understanding
- BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
- Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Constrained Sequence-to-Tree Generation for Hierarchical Text Classification
- Coherent Hierarchical Multi-Label Classification Networks