Many widely used language benchmarks are built from news articles: summarization sets like XSum, event-causality sets like EventStoryLine, and document event extraction sets like DocEE. Large language models have almost certainly read many of these articles during training. Most contamination detectors look for traces of memorized text, such as unusually low loss or near-verbatim regeneration. These traces can be erased: one round of reinforcement learning on clean data, or training on reasoning traces, pushes standard detectors to chance. We propose a behavioral test that does not depend on such traces. Take a benchmark article and rewrite it so that a different event becomes the story's main event, with a new headline, a new lead, and new emphasis. Then ask the model what the article is about. A model that reads the document will follow the rewrite. A model drawing on its memory of the original article will slip back to the old main event.
The probe draws on narrative centrality, the study of which event a document is organized around. That literature gives us the tools to make such rewrites well-defined and verifiable. It also tells us which surface cues (headline, lead, mention frequency) a non-memorizing reader relies on. We set every one of those cues to point at the new main event. Reading the document and reading its heuristics then agree, and only memory disagrees. That makes "reversion to the original main event" a specific signature of contamination. We calibrate it against the same rewrites applied to articles published after the model's training cutoff.
Core Claim
A model that answers from memory will, when asked about a rewritten article, tend to report the main event of the original version. We call this tendency the reversion rate. We claim that the reversion rate on pre-cutoff benchmark articles, minus the reversion rate on matched post-cutoff articles, measures how much the model's benchmark behavior comes from memory overriding the input. This is close to what Magar and Schwartz call exploitation, and it is distinct from memorization. Because the signal is the model's output behavior rather than its token probabilities, we expect it to survive the reinforcement-learning and reasoning-trace "cloaking" that defeats loss-based detectors.
Where This Sits
Three strands of contamination work frame the proposal.
One strand motivates it by showing that probability-based detection is fragile. On The Fragility of Benchmark Contamination Detection in Reasoning Models contaminates models during supervised fine-tuning and then runs one epoch of clean GRPO, a reinforcement-learning method. LiRA's AUROC drops by about 14% from 89.1%, and reference-free detectors fall to about 50%. Under reasoning-trace contamination, ten detectors average 50–58% AUROC while accuracy is inflated by 11.7%. The cause is that these detectors read token probabilities, which post-training reshapes. Our probe reads where the model's answer lands, not how likely the original text is.
A second strand supplies the closest method. Data Contamination Quiz asks a model to recognize the original instance among synonym-swapped variants, and it estimates about 95% contamination of GPT-4 on XSum. Its perturbations are word-level, and it measures recognition rather than whether memory changes task outputs. Our perturbations are structural, and we measure task behavior. Generalization or Memorization detects memorization through collapsed output diversity (CDD) and finds a roughly 41.5% gap on rephrased HumanEval. CDD still keys on surface edit distance, which paraphrase-level leaks can evade.
A third strand constrains the interpretation. Data Contamination: From Memorization to Exploitation shows that BERT can memorize about 15% of injected items while exploiting less than 1%, with exploitation rising to 40% under heavy duplication. A recognition signal can therefore overstate harm. This is why we target the behavioral effect directly.
On the narrative side, Unveiling Causal Reasoning in Large Language Models uses news published after the training cutoff (CausalProbe-2024) to show causal accuracy falling below 70% for Claude 3 Opus. It works at the passage level, ignores document structure, and cannot say which items in an old benchmark are contaminated. We reuse its post-cutoff idea only as a calibration control.
The rewrite machinery comes from the centrality literature. Discourse as a Function of Event defines eight discourse roles organized around a Main Event and reports that Main Event sentences share events with the headline 58% of the time, against 9% for anecdotal sentences. CHEER predicts central events at nearly 80% F1, mostly from early appearance and mention degree. Proximity-Based Entity Ranking warns that leads can be hooks rather than the core of the story. Together these tell us which cues a heuristic reader uses, so that a rewrite can move all of them at once. To our knowledge nobody has used centrality as an axis for perturbing contaminated documents.
Research Program
Building the rewrites. We construct centrality-swapped versions of benchmark articles from XSum, EventStoryLine, NewsDiscourse, and DocEE.
For each article, annotators using the NewsDiscourse role scheme pick a non-main event to promote. Candidates are a Consequence, a Previous Event, or a Current Context event that is plausible as a news anchor. A rewriting model, checked by humans, then rewrites the headline, moves sentences about the promoted event into the lead, and demotes the original main event to background. The rewriter preserves sentences elsewhere where possible, so that most tokens survive.
Two rewrite strengths isolate different readers. In a full swap, headline, lead position, and mention frequency all favor the new main event, so both a heuristic reader and a careful reader should follow it. In a deep swap, the surface cues are left alone and only the causal and temporal framing changes. This tests structural understanding, and it is reported separately rather than used for detection.
Every rewrite is validated by three annotators who have not seen the original. An item is kept only if they identify the intended new main event. This guarantees that a correct reading of the edited text exists.
Building the control set. We apply the same rewrite procedure to articles from the same outlets and genres published after each model's training cutoff. These articles are annotated with the same roles and matched to the benchmark articles on length and genre (disaster, crime, business, politics).
Measuring behavior. For each item we measure behavior on four tasks:
- a one-sentence XSum-style summary,
- a question asking for the main event,
- an EventStoryLine-style causal question about the promoted event,
- DocEE event-type classification, where promoting a different event changes the gold type.
A response counts as reverted when a judge, blind to which version is original, matches it to the original main event rather than the promoted one. Samples of the judge's decisions are audited by humans.
Scoring contamination. Let $R_b$ be the reversion rate on rewritten benchmark articles and $R_f$ the rate on rewritten fresh articles. The contamination score is
$$C = R_b - R_f,$$
with confidence intervals from bootstrapping over articles. Subtracting $R_f$ removes reversion caused by generic world-knowledge priors, for example a model preferring "earthquake" over "road closure" as the main event regardless of memory. For item-level detection, each article gets several independent swaps, and its reversion count is its detection score.
Baselines
The naive reference point is to report benchmark accuracy as is and to report $R_f$ alone. If benchmark reversion is no higher than fresh reversion ($C\approx 0$), the probe adds nothing. We also report the accuracy gap between benchmark and fresh articles without rewriting. That raw gap confounds contamination with time shift and difficulty, and showing this confound is part of the argument.
The strongest existing detectors in the regime we target are LiRA and Loss as evaluated in On The Fragility of Benchmark Contamination Detection. LiRA reaches 89.1% AUROC after supervised contamination, falls by about 14% after one GRPO epoch, and the paper's ten detectors average 50–58% AUROC under reasoning-trace contamination. We will rerun Min-K% Prob, Loss, and LiRA on the same open models and the same seen/unseen splits in our controlled lab. LiRA gets the same reference-model budget the paper describes.
We also run CDD on identical items with the same sampling budget as our probe (about twenty samples per item). The file gives only relative gains for CDD (21.8–30.2% in Accuracy and AUC on DETCON) and no absolute AUROC on news summarization.
DCQ is the strongest black-box competitor and the one aimed most directly at XSum, with an estimated ~95% contamination for GPT-4. We run it with its positional-bias quizzes on the same XSum items. We compare its item-level flags with ours on the controlled models, where ground truth is known. On API models we also check whether its ~95% estimate co-occurs with nonzero behavioral reversion. No reported result was found on Lacuna for any centrality-based or structural-perturbation contamination probe.
Beating the baselines means two things. First, after GRPO and under reasoning-trace contamination, our item-level AUROC must exceed the best probability-based detector by a margin outside bootstrap error. Second, $C$ must track measured exploitation (seen minus unseen accuracy) better than DCQ's recognition rate does.
The key ablation is a paraphrase-only control. It rewrites the article's wording to the same extent as a swap but leaves the main event unchanged, in the spirit of HumanEval_R. If paraphrase-only accuracy drops detect contamination as well as centrality swaps do, then the narrative-centrality ingredient is not doing the work.
Why not just…?
Why not just test on post-cutoff news, as CausalProbe-2024 does? Fresh news shows that a model is weaker on new material. As Unveiling Causal Reasoning itself illustrates, it cannot separate contamination from temporal shift and harder content. It also cannot say which items of an existing benchmark are compromised. We use fresh articles only as the calibration term $R_f$. The raw-gap baseline will show how far fresh-news gaps over- or under-state $C$ on models whose contamination we control.
Why not just paraphrase the benchmark and measure the accuracy drop? This is the real threat, and the paraphrase-only ablation exists to settle it. Our expectation is that paraphrasing leaves the story's events intact. A model that has internalized the story, which is the kind of contamination the Fragility paper shows survives reasoning-trace training, can still answer correctly from memory and show no drop. A centrality swap makes the memorized answer wrong. If paraphrase-only matches our AUROC in the controlled lab, we will report that and recommend the cheaper method.
Why not just use Min-K% or loss thresholds, which are free given logits? On models that received reinforcement learning after contamination, these detectors fall to near chance (Fragility). They are also unavailable for closed APIs. We rerun them anyway: if they recover on news text, the probe is unnecessary for open models.
Why not just rely on lead and headline heuristics as the "non-memorized" baseline? Full swaps move every heuristic cue, so a lead- or headline-following reader is not penalized and cannot cause reversion. Deep swaps, which leave cues in place, are reported separately as a test of structure rather than of memory. This keeps the CHEER-style positional shortcut from being mistaken for contamination.
Proposed Experiments
Controlled lab. We fine-tune open models, including DeepSeek-R1-Distill-Llama-8B as in the Fragility setup, on a random half of XSum and EventStoryLine articles with their annotations. Variants include plain supervised fine-tuning, fine-tuning followed by one epoch of GRPO on clean data, and training on reasoning traces. Duplication levels of 1, 10, and 100 copies probe the duplication dependence that Magar and Schwartz found. Seen and unseen halves give ground truth.
We measure item-level AUROC for our probe, the paraphrase-only ablation, Min-K%, Loss, LiRA, CDD, and DCQ. We also measure the correlation between $C$ and actual exploitation across conditions.
The decisive outcome comes after GRPO and under reasoning-trace contamination. Either centrality-swap AUROC stays clearly above the 50–58% range reported for probability detectors and above paraphrase-only, or it does not, in which case the core claim fails.
Audit of deployed models. We compute $C$ with bootstrap intervals for several open and closed models on XSum, EventStoryLine, NewsDiscourse, and DocEE, each with matched fresh controls. For DocEE we report whether a positive $C$ coincides with the near-human 90.5% event-classification F1 reported in DocEE. That would suggest part of that score reflects memory.
Validity checks. - Human annotators' agreement with the intended new main event must be high on both benchmark and fresh rewrites, so that rewrite difficulty is matched. - The rewriting model is varied, and its own outputs are kept out of evaluation, to rule out rewrite artifacts. - Reversion must rise with duplication count in the lab. A flat curve would mean the probe measures something other than memory.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
- Data Contamination: From Memorization to Exploitation
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Discourse as a Function of Event: Profiling Discourse Structure in News Articles around the Main Event
- CHEER: Centrality-aware High-order Event Reasoning Network for Document-level Event Causality Identification
- Proximity-Based Entity Ranking
- DocEE: A Large-Scale and Fine-grained Benchmark for Document-level Event Extraction