A common way to check whether a language model has memorized a benchmark is to replace the people, places, and things in each question with different ones and see whether accuracy drops. The trouble is that the replacements are almost never equally well known. Fresh news entities, invented names, or random substitutes tend to be obscure, and models are worse on obscure entities whether or not they ever saw the test set. A drop after substitution therefore mixes two things: memorization, and the ordinary penalty for asking about less famous things. This note proposes to remove that confound with a tool from entity linking, the task of mapping a name in text to the right entry in a knowledge graph (a database of entities connected by typed relations). Entity linking research has long measured how well known an entity is through its connectivity in such graphs: link counts, PageRank, and personalized random-walk scores.
The proposal builds, for every entity-centric benchmark item, a "twin" question about a different entity. The twin has the same relation and question template, and the new entity is matched to the original on graph centrality and local neighborhood structure. A model that never saw the benchmark should answer original items and their twins at the same rate. A gap that survives this matching estimates item-specific memorization. Because the test compares answers rather than token probabilities, it should keep working in the setting where probability-based detectors are known to fail, which is after reinforcement-learning post-training.
Core Claim
Graph centrality is a public, model-independent stand-in for how often an entity appeared in pretraining text. Entity linking work shows that it strongly predicts how hard a factual question is. Entity Disambiguation for Knowledge Base Population found Wikipedia PageRank and in- and out-degree to be among the strongest priors for ranking candidate entities. The combination of context information to enhance simple question answering raised Top-1 entity accuracy on SimpleQuestions from 59.5% to 64.0% using Freebase out-degree alone. If centrality predicts difficulty this well, then any contamination test that substitutes entities without controlling centrality is measuring popularity as well as leakage.
We claim three things. First, centrality-matched twins yield an inflation estimate that is near zero on clean models. Second, unmatched substitution produces spurious "contamination" on those same clean models. Third, the matched estimate recovers injected contamination even after a round of clean reinforcement learning. That last setting is where On The Fragility of Benchmark Contamination Detection in Reasoning Models showed log-probability detectors collapsing toward chance, while the accuracy boost (8.8% pass@1) persisted.
Matching also gives a principled answer to the tension between the two literatures. Using a popularity prior is a legitimate skill, and contamination detection should not punish it. A twin with the same centrality leaves that skill fully usable, so only memory of the specific test item can open a gap.
Relation to Prior Work
Two recent systems already substitute or expand entities to fight contamination. Both supply the general move this proposal reuses. CoreEval replaces the factual core of NLU test items with post-cutoff facts from the GDELT news database and reports average drops of 5.42% for proprietary models and 3.62% for open models. Dynamic Multimodal Evaluation via Knowledge-Enhanced Benchmark Evolution rewrites VQA items as triplet subgraphs and adds hops to make them harder. Neither controls the centrality of the substituted entity. CoreEval's fresh entities are new and therefore weakly connected, so its drop cannot be split into memorization and obscurity. KBE-DME raises difficulty on purpose, which is the opposite of what a detector needs.
Two performance-based detectors motivate the statistical design. ConStat compares accuracy on a benchmark against a reference benchmark, using a fleet of models to fit a correction for differing hardness. PaCoST runs a paired test on stated confidence for original versus rephrased questions. Both build their references by LLM rephrasing, and rephrasing keeps the memorized entity–answer pair intact. Our twins change the fact while holding difficulty fixed through an explicit structural covariate, item by item, instead of through a fitted aggregate curve.
RADAR separates recall from reasoning using internal activations. It needs white-box access and was evaluated on 100 curated examples. We need only answers.
The fragility paper is a constraint as well as a motivation. In its second stage, models contaminated with chain-of-thought traces generalize the reasoning to similar unseen questions, so detectors fail there. We restrict scope to single-fact, entity-centric items. In those items there is no transferable procedure, only the fact itself, so a twin cannot borrow the memorized answer.
Research Program
The pipeline has four parts.
The first part is anchoring. Each benchmark item is mapped to a subject entity, a relation, and a gold answer in a reference graph: Freebase FB2M for SimpleQuestions, and the Wikipedia hyperlink graph and Wikidata for the other benchmarks.
The second part is the centrality profile. For each entity we compute log in-degree, global PageRank, relation-specific out-degree (how many objects it has for this relation, which captures answer ambiguity), and the personalized PageRank mass from subject to answer. The personalized PageRank score follows Combining Mention Context and Hyperlinks from Wikipedia for Named Entity Disambiguation, which used random walks that restart at candidate entities to score coherence. It captures how strongly the answer is associated with the subject in the graph.
The third part is twin construction. For each item we sample a substitute subject of the same type that has the same relation, a single unambiguous gold object, and a profile within a fixed tolerance of the original. We fill the item's original surface template with the substitute's name. Wording is held constant, so rephrasing is not a variable. A subsample is checked by hand for answerability and label correctness.
The fourth part is the inflation estimate. For a model $M$ and items $i = 1, \dots, n$ with twins $\tilde{x}_i$, we define
$$\hat{\delta}_M = \frac{1}{n}\sum_{i=1}^{n}\left[\mathbb{1}\{M(x_i) = y_i\} - \mathbb{1}\{M(\tilde{x}_i) = \tilde{y}_i\}\right],$$
where $x_i$ and $y_i$ are the original question and answer, and $\tilde{y}_i$ is the twin's gold answer. Significance comes from a paired permutation test, which needs no fixed threshold. We also run a regression variant for cases where exact matching is too strict: accuracy on twins is modeled as a function of the centrality profile, and the original items are tested for excess accuracy given their profiles. This is item-level hardness correction in the spirit of ConStat, with an observed structural covariate in place of a fitted curve.
The analysis has one key question. Which components of the profile are actually needed? We answer it by matching on increasingly rich subsets and checking which subset brings clean-model $\hat{\delta}$ to zero.
Baselines
Naive substitution. Replace each subject with a random same-type entity that has the same relation, with no centrality control. This is the "do nothing about the confound" reference. We expect it to show a positive $\hat{\delta}$ on clean models, and the size of that spurious gap is itself a result. A twin set that passes the clean-model calibration while this one fails counts as beating it.
Fresh-entity substitution in the style of CoreEval. Substitute subjects drawn from entities created after the model's cutoff. This is the closest existing method and the most direct competitor. We will run it on the same SimpleQuestions items, with the same models and the same hand-verification budget as our twins. No detection AUROC is reported for CoreEval; the file gives only its aggregate drops of 5.42% and 3.62%. We beat it if its clean-model gap is significantly nonzero while ours is not, and if our estimate tracks injected contamination more closely.
ConStat. This is the strongest performance-based detector. It reports detecting 89% of syntax-contaminated and 98% of sample-contaminated models, and it found δ ≈ 8.2% for Mistral-7b-v0.1 on GSM8k. We run it with its rephrased reference benchmarks and its own fleet of reference models, and we give our method the same reference models for calibration. We beat it if we match its detection rate on plain fine-tuning contamination and exceed it on contamination with rephrased question text or after reinforcement learning. Rephrased contamination is the case where its reference set and the contaminated data share the memorized fact.
Probability-based detectors: LiRA, Min-K% Prob, and PaCoST. On contamination by supervised fine-tuning, the fragility paper reports LiRA at 89.1% AUROC, falling by about 14% after one epoch of GRPO, with reference-free methods near 50%. No reported detection AUROC was found on Lacuna for PaCoST; it reports only significance at p < 0.05. All of these run on the same checkpoints. We beat them if our AUROC holds after GRPO while theirs falls.
Ablation of the key ingredient. Profiles are matched on nothing, then degree only, then degree plus PageRank, then the full profile including relation out-degree and personalized PageRank. This shows whether graph structure is doing the work. It is also compared against matching on surface-string frequency in an open corpus.
Why not just…?
Why not just rephrase the questions, as PaCoST and ConStat do? Rephrasing leaves the memorized subject–answer pair in place, and memorized facts survive paraphrase. The fragility paper shows detectors falling to 50–58% AUROC when models generalize to similar unseen phrasings. Our experiment with rephrased-text contamination tests this directly: if rephrasing-based references detect it as well as twins do, this argument fails.
Why not just use post-cutoff entities? Post-cutoff entities are obscure by construction, and the SimpleQuestions out-degree result shows that obscurity alone moves accuracy by several points. The clean-model calibration settles the question. If fresh-entity substitution shows no gap on clean models, then the confound is small in practice and our matching adds little.
Why not just match on how often the entity's name appears in text, or on page views, instead of graph structure? This is the real threat. Snoopy ties few-shot accuracy to pretraining term frequency, and string frequency may be as good a proxy for exposure as degree. The ablation includes frequency matching. If it ties the graph profile, we will report that the useful contribution is matched-twin testing itself, with graph centrality as only one sufficient covariate. Two graph features have no string-frequency analogue: relation-specific out-degree and subject-to-answer personalized PageRank. Our prediction is that these explain residual gaps that remain after frequency matching.
Why not just use probability-based detectors? They need white-box scores. Reinforcement learning with clipped updates shrinks the probability gap they depend on, while the accuracy gain survives. A detector that reads accuracy directly is aimed at the quantity that persists.
Proposed Experiments
The primary benchmark is SimpleQuestions over FB2M. Every item there is a single triple, so twins can be generated exactly with gold answers from the graph. We extend to ConvQuestions, using the first turn only to keep items single-fact, and to entity-linking items from AIDA-CoNLL and TAC-KBP posed to LLMs as multiple choice.
The experiments use controlled contamination. Open models from the families examined in PaCoST, such as Llama and Qwen, are fine-tuned on a random half of the test items. We vary the dose, use verbatim versus rephrased text, and then optionally apply one epoch of clean GRPO, following the fragility paper's protocol. The untouched half and unmodified checkpoints serve as clean references.
There are three decisive checks.
The first is calibration. On clean checkpoints, matched $\hat{\delta}$ should not differ significantly from zero, while naive substitution and fresh-entity substitution should show positive gaps that grow with the centrality mismatch.
The second is recovery. On contaminated checkpoints, $\hat{\delta}$ should track the injected accuracy gain measured on the contaminated half. We report correlation with dose and item-level AUROC, and we require it to beat LiRA after GRPO.
The third is attribution. The profile ablation should show that clean-model $\hat{\delta}$ reaches zero only once relation out-degree and personalized PageRank are matched, rather than with string frequency alone.
If calibration fails, meaning even fully matched twins show gaps on clean models, then centrality does not capture difficulty well enough for this use, and that negative result would itself be worth reporting. Finally, we apply the audit to released open models on SimpleQuestions and the entity-centric parts of TruthfulQA and MMLU, comparing our flags with PaCoST's and ConStat's.
References
- Entity Disambiguation for Knowledge Base Population
- The Combination of Context Information to Enhance Simple Question Answering
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- CoreEval: Automatically Building Contamination-Resilient Datasets with Real-World Knowledge toward Reliable LLM Evaluation
- Dynamic Multimodal Evaluation via Knowledge-Enhanced Benchmark Evolution
- ConStat: Performance-Based Contamination Detection in Large Language Models
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation
- Combining Mention Context and Hyperlinks from Wikipedia for Named Entity Disambiguation
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance