When a language model scores well on a document-level relation benchmark, we cannot tell whether it read the document or remembered it. Suppose the benchmark asks whether a chemical causes a disease, based on a PubMed abstract. The model may have seen that abstract, or its label, during pretraining. Detectors that look for this, by checking whether the model assigns unusually high probability to benchmark text, are now known to fail on models trained with reinforcement learning or on chain-of-thought data. We propose a behavioral test that does not look at probabilities at all. We take the few sentences a relation actually depends on and change only their small words. Adding "not", reversing "caused by" into "causes", or changing a pronoun so it can no longer refer to the entity is enough to change the correct answer. A model that reads will follow the edit. A model that recalls will keep giving the old, now wrong, label. We call the second behavior recall override.
The test has two ingredients. The first is a heterogeneous document graph, meaning a representation of the document with typed nodes (entity mentions, entities, sentences) and typed edges (co-occurrence, coreference, bridge links through a shared entity). It tells us which sentences lie on the reasoning path for a given relation, so we can edit on-path sentences and leave off-path ones as controls. The second comes from work on short texts, which shows that function words, casing and punctuation carry polarity, direction and certainty. That is exactly the signal that standard preprocessing strips out and that surface-level contamination checks ignore. The contribution is a way to measure whether a model's answers on a static document benchmark depend on the document's evidence path or on memory of it. The measure is designed to keep working in the regime where loss-based detectors do not.
Core Claim
Minimal, label-changing edits to the function words of on-path evidence sentences produce a contamination signal that is local, directional and hard to cloak. The key quantity is how often the model keeps the pre-edit label on familiar documents compared with the same edit on documents whose entities have been renamed to fictional ones. For a document set $D$, let $\mathrm{ROR}(D)$ be the recall override rate, the fraction of label-flipping edits on which the model still outputs the original gold label. The contamination gap is
$$G = \mathrm{ROR}(D_{\text{orig}}) - \mathrm{ROR}(D_{\text{renamed}}),$$
where $D_{\text{renamed}}$ is the same documents with every entity consistently replaced by an invented name and the identical function-word edit applied. A model with no memory of the documents should have $G \approx 0$: renaming removes recall but leaves the reading problem unchanged. A model that memorized the documents should show $G > 0$, because recall can only override the edit when the original entities are present. We predict that $G$ stays clearly positive after reinforcement learning erases the loss gap between seen and unseen data. The reason is that such training changes the model's likelihoods, while $G$ is measured on the model's discrete answers under a counterfactual. The original-document side of that counterfactual exists only if the text was memorized.
Grounding in Prior Work
Three findings motivate the proposal. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that reinforcement learning with clipped importance sampling, such as GRPO, collapses the log-probability gap that Loss, Min-K% and LiRA rely on. After contamination with chain-of-thought data, all ten detectors it tests fall to 50–58% area under the ROC curve (AUROC). Metadata Predictability Is Not Evidence Dependence shuffles and permutes whole passages on HotpotQA, FEVER and SNLI. It finds that models can ignore evidence entirely, with an evidence-intervention effect near zero on HotpotQA. Its interventions work at the passage level, so they cannot say which edge of the reasoning chain a model skips. Discriminative Reasoning for Document-level Relation Extraction splits DocRED relations into intra-sentence, logical-bridge and coreference paths. It reports roughly 80% F1 intra-sentence but 38% on logical and 44% on coreference reasoning. This suggests high aggregate scores may rest on patterns other than multi-hop reading.
The method draws on graph work. Inter-sentence Relation Extraction with Document-level Graph Convolutional Neural Network, Connecting the Dots and Document-level Relation Extraction with Dual-tier Heterogeneous Graph build mention–entity–sentence graphs with coreference and sentence edges as shortcuts between distant evidence. They use these graphs to predict relations. We use the same node and edge types to decide where to intervene. We also avoid their reliance on external parsers, a known source of error cascades, by building graphs only from the gold mention and evidence annotations that DocRED (DocRED) and CDR supply.
The edit design draws on short-text work. CU: Computational Assessment of Short Free Text Answers showed that words like "it" and "was" carry the meaning of one- or two-sentence answers. I2C at SemEval-2022 Task 6 found that cased models that keep punctuation outperform uncased ones on sarcasm. The ICTNET microblog studies (2013, 2014; ICTNET at Microblog Track in TREC 2014) found that keeping stopwords improves retrieval. We treat each evidence sentence as a short text of this kind, where a single function word can decide the label.
The closest dynamic-evaluation work is DyVal and DARG. Both generate or perturb reasoning graphs to escape contamination, and DARG reports over 50% drops for Claude-3-Opus under deeper graphs. Their graphs are symbolic arithmetic and logic structures, and their perturbations make problems harder, so a score drop mixes contamination with difficulty. We change neither difficulty nor domain. We make minimal edits to real natural-language documents and read the direction of the error, not only its size.
Research Program
The first component is an edit catalogue tied to edge types. For intra-sentence relations, the edits are negation insertion ("was associated with" becomes "was not associated with") and direction reversal through voice or preposition ("induced by" versus "induces"). For coreference paths, the edit changes a pronoun's number or gender so that the coreference edge no longer resolves, which makes the relation unsupported. For bridge paths, the edit negates or hedges the relation in one bridge sentence, which breaks the chain. Each edit changes at most three function tokens and leaves every content word and entity string unchanged. An LLM proposes candidate edits from the catalogue. Two annotators then confirm the new gold label: flipped, reversed, or no-relation.
Two controls go with every edit. An off-path control applies an equivalent edit to a sentence that is not on the evidence path, where the answer should not change. A renamed twin applies the same edit to the document after consistent fictional entity substitution. Together these give a factorial design: original versus renamed, on-path versus off-path, and edit type by reasoning type.
The second component is a set of documents where we know the contamination status. There are three regimes. First, supervised document relation models such as ATLOP and the reasoning network above have seen DocRED training documents with labels. Scoring them on edited training documents against edited test documents gives label-level contamination with certain membership at essentially no cost. Second, we contaminate open models under our control. Following the two-stage protocol of the fragility paper, we fine-tune on a random half of the DocRED and CDR test documents, then apply one epoch of GRPO. This gives known members and non-members before and after cloaking. Third, for open models trained on FineWeb, we split benchmark documents by n-gram overlap with the corpus into likely-seen and likely-unseen sets. This is a noisier label for natural contamination.
The third component is analysis. We report $\mathrm{ROR}$ and $G$ per edge type. Because each document can carry several independent edits, we also aggregate a per-document contamination score, and use it for membership AUROC so it can be compared directly with the detectors.
Baselines
The naive reference point is plain benchmark accuracy on unedited documents. This is the number contamination inflates. Published figures set its scale: 61.37 F1 on DocRED for the BERT-base discriminative reasoning network and 61.30 F1 for ATLOP-BERT, both reported in Discriminative Reasoning for Document-level Relation Extraction, and 65.9% overall F1 on CDR for DHG-BERT in Dual-tier Heterogeneous Graph. We will rerun these rather than quote them, because edited-set scores must use the same checkpoints.
The strongest existing detection competitor is LiRA, a reference-model likelihood-ratio membership test. On The Fragility of Benchmark Contamination Detection in Reasoning Models reports that it reaches 89.1% AUROC after contaminated fine-tuning and loses about 14 points after one GRPO epoch. We will also run Loss and Min-K%. We will run CDD from Generalization or Memorization, which flags memorization when sampled outputs cluster tightly by edit distance; that paper reports relative gains of 21.8–30.2% over earlier detectors. All detectors get the same member and non-member documents, the same model checkpoints, and the same reference model where one is needed. Thresholds are tuned on the same held-out split used for our score. Our method beats them if its membership AUROC is at least as high before cloaking and stays above 75% after GRPO, where we expect likelihood detectors to fall toward the 50–58% range reported for the chain-of-thought stage.
The strongest behavioral competitor is entity renaming alone: measure accuracy on documents with fictional entities and treat the drop as contamination. We found no reported result on Lacuna for entity renaming on DocRED or CDR, so we will build it ourselves with the same renamed documents our method uses. The second behavioral competitor is the passage-level evidence intervention from Metadata Predictability Is Not Evidence Dependence, applied to DocRED sentences and HotpotQA supporting paragraphs.
The key ablation removes the graph. Function-word edits go to randomly chosen sentences instead of graph-identified on-path sentences, with the same edit budget. If the graph adds nothing, $G$ computed this way should match; we predict it will be diluted by off-path edits that should not change the answer. A second ablation replaces function-word edits with content-word substitutions of the same length, to test whether the scaffold edits are special.
Why not just…?
Why not just rename the entities and measure the accuracy drop? This is the most serious alternative, and it is part of our design. The weakness of renaming alone is that its drop has two causes. Contamination is one. The other is that invented names are harder for any model: they tokenize differently and lack world-knowledge support, even for a clean model. Renaming therefore also drops on documents we know are unseen. The minimal-pair edit separates the two causes. Both arms receive the same edit, so the difficulty of unfamiliar names cancels in $G$. The direction of the error then matters: keeping the old gold label is evidence of recall, whereas a random error is not. The known-membership regime settles this. If renaming-drop AUROC matches the AUROC of $G$ on the controlled GRPO models, the simpler test is enough and we will say so.
Why not just paraphrase the benchmark, as EvoEval does? The survey Benchmark Data Contamination of Large Language Models reports a 39.4% average drop for 51 models on evolved HumanEval. That shows paraphrasing exposes inflation. But Generalization or Memorization shows that rephrased leaks still cause memorized behavior. Paraphrasing also keeps the label fixed, so a model can answer correctly from recall of the entity pair. Our edit changes the label while keeping the entities and nearly all tokens, so recall and reading give different answers.
Why not just use likelihood detectors or CDD? Likelihood detectors are exactly what reinforcement learning cloaks, per the fragility paper. CDD depends on sampled outputs collapsing onto a memorized string. That applies to generation tasks, but it is unclear how it applies to a relation label drawn from a small closed set, where outputs cluster for clean models too. The head-to-head on cloaked checkpoints tests both.
Why not just shuffle or remove evidence passages? Passage removal shows whether the model uses the evidence at all. It cannot show which link is skipped, and removing a passage also removes entity mentions, which changes the question. Edge-typed edits say, for example, "this model follows intra-sentence negation but ignores broken coreference". That per-path diagnosis is what the reasoning network's 38%/44%/80% split calls for.
Proposed Experiments and Decisive Checks
The first experiment is a sanity check with known membership. ATLOP and the reasoning network are scored on edited DocRED training documents against edited test documents. We expect $G$ to be clearly positive on training documents and near zero on test documents. If $G$ does not separate these, the method fails at its easiest case.
The second experiment is the cloaking test. Using open LLMs contaminated on half of the DocRED and CDR test documents, we compare the membership AUROC of $G$ with LiRA, Loss, Min-K% and CDD before and after one GRPO epoch, and after chain-of-thought contamination. The claim fails if $G$ loses as much AUROC as LiRA.
The third experiment is the per-path profile. On the same models, we report recall override separately for intra-sentence, coreference and bridge edits. This tests the prediction that recall dominates most on the multi-hop paths where honest reading is hardest.
The fourth experiment applies the audit to HotpotQA supporting facts and to FineWeb-trained open models. It gives a first estimate of natural contamination on these document benchmarks.
We will release the edited and renamed document sets with annotator-verified labels, together with the edit catalogue. Because the edits are generated per document and can be resampled, a new audit set can be produced after any leak.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Metadata Predictability Is Not Evidence Dependence: An Intervention-Based Audit for Weak-Label Benchmarks
- Discriminative Reasoning for Document-level Relation Extraction
- Inter-sentence Relation Extraction with Document-level Graph Convolutional Neural Network
- Connecting the Dots: Document-level Neural Relation Extraction with Edge-oriented Graphs
- Document-level Relation Extraction with Dual-tier Heterogeneous Graph
- DocRED: A Large-Scale Document-Level Relation Extraction Dataset
- CU : Computational Assessment of Short Free Text Answers - A Tool for Evaluating Students' Understanding
- I2C at SemEval-2022 Task 6: Intended Sarcasm Detection on Social Networks with Deep Learning
- ICTNET at Microblog Track in TREC 2013
- ICTNET at Microblog Track in TREC 2014
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
- Benchmark Data Contamination of Large Language Models: A Survey