A language model that scores well on a public benchmark may have learned the skill, or it may have seen the test questions during pretraining and memorized them. This proposal tests the difference by moving each benchmark question sideways in a concept hierarchy. A concept hierarchy is a hand-built tree of "is-a" relations such as MeSH, the medical subject tree, or WordNet. For example, "migraine" becomes "cluster headache," a sibling under the same parent, "headache disorder." A model that understands the category should mostly survive the swap, losing accuracy gradually as the substitute moves further away in the tree. A model that memorized the item should fail sharply at the first swap, or keep giving the memorized answer when the swap has changed the correct one.
The idea borrows from the vision-language finding that contaminated models collapse when the target concept in an image is edited. That work reports a score change of −45% for contaminated models against +31.5% for clean ones on MMStar (Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations). We carry this logic to pure-text benchmarks. Instead of pixel edits, the perturbations follow structured lexical hierarchies, and their size is set by distance in the tree.
Core Claim
Contamination leaves a specific signature in how accuracy falls with distance in the concept tree, once term frequency is held fixed. A clean model's accuracy falls smoothly as the substitute moves from sibling to cousin to distant relative. A contaminated model shows a step: a large drop at the smallest swap, then a flatter curve. On items where the swap changes the correct answer, it also keeps producing the original answer at a high rate. We call this rate "memorized-answer persistence." The claim is that the step and the persistence rate together detect contamination, including paraphrased contamination, better than plain rewording does. The test needs only black-box access and a handful of queries per item.
Two strands of prior work motivate this expectation. Classifying the Semantic Relations in Noun Compounds via a Domain-Specific Lexical Hierarchy showed that a word-level classifier collapses to about 8% accuracy on unseen biomedical words. A classifier using MeSH ancestors keeps 38–40% on the same words. Category-level knowledge therefore survives swaps that defeat item-specific knowledge, and that asymmetry is what we exploit. Think or Remember? found that memorization and generalization separate inside small models on synthetic minimal pairs. Our sibling swaps are minimal pairs drawn from real ontologies and applied to real benchmarks, read out from behavior rather than from neurons.
Research Program
The method builds a perturbation generator that maps the entities in each benchmark item to an ontology. We use MeSH and the UMLS Semantic Network for medical and biology items, following the two-tier construction in GEANet, and WordNet for general items. For each mapped entity the generator produces substitutes at tree distance 2 (sibling), 4 (cousin) and 6 or more.
Each substitute must lie within a fixed band of the original's corpus frequency. We estimate frequency on The Pile, following Snoopy, which showed that few-shot accuracy tracks pretraining term frequency. Without this matching, a clean model would also drop on swaps to rare terms, and the test would confuse rarity with memorization.
Substitutions come in two types. Answer-invariant swaps replace an entity whose identity does not decide the answer, for example a WinoGrande object replaced with a sibling that has the same relevant property. Answer-tracking swaps change the gold answer in a way the ontology or a verifier can determine, which allows persistence to be measured. Climbing to a parent concept can strip the context the answer depends on, a failure mode that CAT documents for conceptualization. For that reason every perturbed item passes an automatic validity check, and a stratified 10% sample is verified by annotators. Items that fail are dropped rather than repaired.
Because real contamination status is unknown for most models, the main evaluation uses controlled contamination with known ground truth. We continue-pretrain open models at roughly 1B and 7B parameters. These sizes straddle the "memorization-limited" and "contamination-exploitable" regimes reported by DCR. The continued-pretraining data is a FineWeb sample (HuggingFaceFW/fineweb) into which held-out benchmark subsets are mixed under the contamination types defined in A Taxonomy for Data Contamination in Large Language Models: verbatim, noised by paraphrase, and label-masked. Each benchmark is split by item into contaminated and clean halves, so detection is scored item by item against known labels.
Baselines
The naive reference is the raw accuracy gap between the contaminated and clean halves, which assumes access to the contamination labels. It is an upper bound on what any label-free test could reveal. We also report it as accuracy on the unperturbed benchmark, which is what a practitioner sees by default.
The strongest existing black-box detector is CDD from Generalization or Memorization. CDD samples many outputs per prompt and flags items whose samples sit at near-zero edit distance from one another. The paper reports relative improvements of 21.8% to 30.2% in accuracy and AUC over n-gram and model-based baselines. It is the closest competitor because, like our test, it needs no corpus access. We run it with its published sampling settings on the same items and models. Our test receives a matched query budget, counting each sampled completion as one query.
The strongest log-probability detector is Min-K% Prob, used as a filter in C2LEVA and CausalProbe-2024. We run it on the open models where log-probabilities are available. We found no reported detection-AUC result on Lacuna for Min-K% Prob on these benchmarks, so we will measure it ourselves.
The paraphrase baseline rewrites each item without changing any entity, in the style of HumanEval_R. On HumanEval_R, Dong et al. report an estimated leak ratio of about 41.5% for ChatGPT.
The key ablation replaces taxonomic substitutes with random frequency-matched entities of the same broad type, with no tree structure. This isolates whether the ontology adds anything beyond "change a noun."
Our test beats the field if three conditions hold. First, its item-level AUC exceeds CDD's on the noised-contamination condition at equal query budget. Second, it exceeds the paraphrase baseline by a margin that holds across both model sizes. Third, the ontology-based version beats the random-swap ablation on the clean-model false-positive rate. The false-positive rate is where tree structure should matter, because random swaps make clean models look contaminated.
Why not just…?
Why not just paraphrase the questions? The contamination taxonomy paper found that training on noised, paraphrased test data boosts scores nearly as much as verbatim exposure. A model contaminated with paraphrases may therefore handle a further paraphrase without trouble. A sibling swap changes the content of the item rather than its wording, so it should still separate such models. This is a genuine threat. If the paraphrase baseline matches our AUC under noised contamination, the hierarchy contributes nothing, and we will report that outcome.
Why not just swap in random entities? Random swaps often break the item or change its difficulty unpredictably. Clean models would then drop as well, and the test would produce false positives. The tree gives a graded distance to read a curve from, and it keeps answer-relevant properties within a category. The random-swap ablation measures this directly. If random swaps match ontology swaps on both AUC and false-positive rate, the tree is unnecessary, and we will say so.
Why not just use CDD or Min-K% Prob? Min-K% needs log-probabilities, which frontier APIs do not expose. CDD needs many high-temperature samples per item, which the contamination survey flags as costly at audit scale. A swap test also tells you how far a model's competence extends, not only whether an item leaked. CDD may still win on verbatim contamination, and we expect it might. Our claim rests on the paraphrased condition and on cost.
Why not just build fresh post-cutoff benchmarks? Fresh data, as in CausalProbe-2024, is the cleanest solution, but it is expensive to repeat and cannot audit the existing benchmarks people still report. Dynamic generators such as DyVal are limited to formal tasks. We use CausalProbe-2024 as a clean control. Every model should show a smooth curve on it, and if one does not, the test is miscalibrated.
Proposed Experiments
The benchmarks are the medicine and biology subsets of MMLU, where MeSH coverage is high, and the MedLine noun-compound set from Rosario and Hearst. We add WSC and WinoGrande, mapped through WordNet, using the artifact-controlled variants from Back to Square One. CausalProbe-2024 serves as the clean control. The primary metrics are item-level detection AUC against controlled-contamination labels, memorized-answer persistence on answer-tracking swaps, and the size of the step at tree distance 2. The step is measured relative to the slope over the larger distances.
The first decisive check runs on the noised-contamination condition, which is where existing tests are weakest. If the sibling-swap AUC does not exceed both the paraphrase baseline and CDD there, the core claim fails.
The second check uses clean models and looks for a smooth, frequency-controlled decline with no step. A step in clean models would mean we are measuring rarity or ontology quirks rather than memorization.
The third check asks whether the step size grows with contamination dose, measured in epochs of exposure, and with model size. Such a dose-response would mirror the LLaVA result, where three epochs of contamination raised the original MMStar score from about 38% to 54% and dropped the perturbed score from about 69% to 38%. Only after these checks pass do we apply the test to public open models with unknown training data, and we label those results as estimates rather than verdicts.
References
- Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations
- Classifying the Semantic Relations in Noun Compounds via a Domain-Specific Lexical Hierarchy
- Think or Remember? Detecting and Directing LLMs Towards Memorization or Generalization
- Biomedical Event Extraction on Graph Edge-conditioned Attention Networks with Hierarchical Knowledge Graphs
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning
- DCR: Quantifying Data Contamination in LLMs Evaluation
- A Taxonomy for Data Contamination in Large Language Models
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
- C2LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Benchmark Data Contamination of Large Language Models: A Survey
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Measuring Massive Multitask Language Understanding
- Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema