Many summarization benchmarks contain reference summaries that state facts the source document never mentions. Over 70% of XSum references contain such facts, as does roughly 35% of Gigaword. This is usually treated as a defect: it teaches models to hallucinate, and it rewards hallucination under overlap metrics like ROUGE. We argue the same defect is a free tool for auditing closed models. If a summarizer's output reproduces exactly the part of a reference that cannot be derived from the source, such as a player's age, a prior club, or a date absent from the article, the most likely explanation is that the model saw that reference during training. Unsupported reference content works like a canary string: a marker that was never planted deliberately but is built into the benchmark.
The opportunity matters because the companies that build the strongest models do not disclose their training data. Audits that do not need that data either depend on following instructions on artificial cloze prompts or need access to token probabilities. Our audit runs on the model's ordinary summarization output, which any API returns. It also answers a question the noisy-reference literature leaves open. When a model scores well on XSum, how much of its ROUGE comes from reproducing ungrounded reference n-grams it has memorized, and how much from summarizing well?
Core Claim
For each benchmark item we split the reference summary into source-grounded spans and source-ungrounded spans, using entity and entailment checks. We then measure an ungrounded reproduction rate: the share of ungrounded reference spans that appear verbatim in the model's summary. We subtract a guessability floor, estimated on matched items the model cannot have seen. The claim is that this corrected rate separates contaminated from clean models and items far better than the same statistic on grounded spans. Grounded spans can be copied from the source by any competent summarizer, so they carry no membership signal. The ungrounded-only restriction is the new ingredient. It turns the hallucination problem documented by the noisy-reference literature into a membership test that needs no training data and no log probabilities.
Relation to Prior Work
Two lines of work motivate the idea. The first finds that reference noise is pervasive and harmful. Questioning the Validity of Summarization Datasets measures the XSum inconsistency rate and filters 42–48% of data with BERTScore, BARTScore and Dependency Arc Entailment. Loss Truncation reports the Gigaword hallucination rate. Training Dynamics for Text Summarization Models shows models learn to copy early and to imitate unsupported details later. Analyzing and Evaluating Faithfulness in Dialogue Summarization finds 17% of SAMSum references inconsistent. All four treat ungrounded spans as noise to remove. We treat them as a measurement instrument. We also reuse their tools, entity matching from Fine-Grained Loss Truncation and entailment from SummFC, to locate those spans.
The second line supplies the auditing logic. Investigating Data Contamination in Modern Benchmarks introduces TS-Guessing, which masks an arbitrary wrong option in a multiple-choice question and checks whether the model can restore it. It reports 57% exact match for GPT-4 on MMLU, but it notes the method depends on the model following instructions. Speak, Memory masks character names in novel passages whose context gives no clue to the name. Both methods rest on the principle we use: content that cannot be inferred, when correctly produced, indicates the model has seen it. Neither method uses the generation task itself, and neither selects its masked slots by checking which content the source actually supports. In summarization, that check is what separates "inferable" from "memorized."
One paper constrains the design. Measuring Non-Adversarial Reproduction shows that comparing outputs against a 10TB proxy web crawl gives only a loose lower bound on memorization, which is why we do not rely on crawl matching.
Liquid-liquid phase separation enters as the evaluation domain, not as a mechanism. We do not pursue the analogy between condensate saturation thresholds and contamination thresholds; it would be decorative. The field has a fast-growing literature full of specific protein names, which makes its unsupported references hard to guess. Examples of such papers include TIFA-TRAF6 condensation, ALYREF phase separation in cardiomyocytes, and paraspeckle-mediated DNA repair. Papers published after a model's training cutoff give a same-genre set of items the model cannot have seen, which we use to measure how often unsupported content is guessed by chance or from general knowledge.
Research Program
The first component is span labeling. For XSum, Gigaword, SAMSum and CNN/DM, we mark a reference span as ungrounded when it meets either test. Under the entity test, it contains a named entity, number, or date absent from the source after normalization. Under the entailment test, Dependency Arc Entailment judges the corresponding arc as not entailed. A 500-item human audit per dataset estimates labeling precision.
CNN/DM has few ungrounded spans, so it serves as a low-power control: the method should have little to say there.
The second component is the guessability floor. A span about a famous footballer's age may be guessable from world knowledge. We estimate this floor in two ways.
The first estimate builds post-cutoff pairs in matching styles. One set is BBC-style news-to-headline pairs collected after the cutoff. The other set comes from recent LLPS papers: the source is the abstract body and the reference is the highlight or title sentence, which often names proteins or mechanisms absent from the abstract. We then measure the ungrounded reproduction rate on these pairs.
The second estimate prompts the model with the source plus an explicit request to infer missing background facts. This gives a knowledge-only upper bound on guessing.
The corrected statistic is the observed rate minus the floor, with bootstrap intervals.
The third component is ground truth. Closed models offer none, so we build it by continued pretraining of open models. The candidates are the Llama 2 and Llama 3.1 families named in the file, together with BART and Flan-T5 as summarizers. We inject randomly chosen subsets of XSum and SAMSum test references at doses of zero, one, four and sixteen copies, both as raw text and as paraphrases. This yields item-level labels for "seen" versus "unseen" and a dose-response curve. It lets us test the contamination survey's claim of non-linear score jumps from small leakage (Contamination Detection Survey).
The fourth component asks whether noise-robust training hides contamination. Fine LT and Error Norm Truncation both drop high-loss tokens. Unsupported spans are exactly such tokens. A model trained with truncation on contaminated data might therefore memorize the grounded parts of leaked references while refusing to emit the ungrounded parts, which would blind our detector. We test this directly by repeating the injection experiment under each truncation objective.
Baselines
The naive reference point is the raw ROUGE-2 gap between a model's XSum score and its score on the post-cutoff BBC-style set. We run it on identical models and prompts. Beating it means our statistic achieves higher item-level AUROC on the injection ground truth and ranks contamination doses correctly where the ROUGE gap does not. The ROUGE gap confounds memorization with distribution shift, which is its expected failure.
The strongest black-box competitor is TS-Guessing (Deng et al.), adapted to summarization: we mask a randomly chosen reference span and ask the model to restore it. Its reported result is GPT-4 at 57% exact match and Llama 2-13B near 0% on MMLU distractor guessing. No reported result was found on Lacuna for TS-Guessing on summarization. We give it the same items, the same models, and the same prompt-tuning budget of 20 templates selected on a held-out dev split. Beating it means higher AUROC on injected open models and smaller dependence on instruction tuning, measured by comparing base and instruction-tuned versions of the same model.
The strongest gray-box competitor is Min-K% Prob and Min-K%++. These methods score the reference by its lowest-probability tokens, and the survey reports gains of up to 10 ROC-AUC points over perplexity. No reported absolute AUROC was found on Lacuna for summarization references. We run it only on open models with full log probabilities. That is an advantage for Min-K%, so matching it while using text alone counts as success.
The key ablation applies our identical pipeline to grounded spans only. If the grounded-span statistic separates seen from unseen items as well as the ungrounded one, our central claim fails.
Why not just…?
Why not just match outputs against a public web crawl? Crawl matching assumes the auditor holds a copy of the training data. Aerni et al. show that even a 10TB proxy crawl gives only a loose lower bound on reproduction (Measuring Non-Adversarial Reproduction). It also cannot tell whether a benchmark item leaked through a mirror the auditor never indexed. We include crawl overlap against public corpora as a secondary signal, but it cannot settle membership for closed models.
Why not just use Min-K% Prob? On open models it may well match or beat us; that is a real threat and the injection experiment will show it. But closed APIs restrict or mask log probabilities, as the survey notes, so Min-K% cannot be applied to the models whose claims most need auditing. The useful comparison is whether our text-only statistic comes within a few AUROC points of Min-K% where both can be run.
Why not just run TS-Guessing or name cloze on reference summaries? This is the closest alternative and the most serious threat. If cloze on random reference spans works as well as our method, the ungrounded restriction adds nothing. We expect it to fall short for two reasons. Random spans are often grounded, so a good summarizer fills them from the source and inflates false positives. And cloze prompts carry the instruction-following confound Deng et al. themselves report. The grounded-versus-ungrounded ablation and the comparison of base and instruction-tuned models settle this directly. We also run a hybrid, cloze restricted to ungrounded spans, to separate the effect of span selection from the effect of free generation versus cloze.
Why not just filter the benchmark, as SummFC does, and stop worrying? Filtering produces a cleaner training and evaluation set going forward. It does not reveal whether a deployed model's past XSum numbers were inflated by memorization, which is the auditing question here.
Proposed Experiments
The decisive experiment is on injected open models. We report item-level AUROC for seen versus unseen references at each dose for four statistics: the ungrounded reproduction rate, its grounded-span counterpart, adapted TS-Guessing, and Min-K%. We expect four results. The ungrounded statistic should beat the grounded one by a wide margin. It should exceed TS-Guessing on base models. It should approach Min-K%. And its signal on CNN/DM should be weaker than on XSum, roughly in proportion to how many ungrounded spans each dataset has.
The paraphrase-injection condition tests robustness. Paraphrasing tends to preserve unsupported facts while changing the surrounding wording, so we expect the ungrounded statistic to degrade less than verbatim n-gram methods.
The truncation experiment reports whether Fine LT or Error Norm Truncation suppresses the signal under contamination. If it does, this is an important caveat: robust training can launder leaked data. We report that finding either way.
Finally, we apply the calibrated statistic to closed models, including GPT-4, Claude 3 and Gemini 1.5, on XSum, Gigaword and SAMSum. We report two quantities. One is the corrected ungrounded reproduction rate with intervals derived from the post-cutoff nulls, including the LLPS null. The other is a ROUGE decomposition into grounded and ungrounded matches, which estimates how much of each model's reported score depends on reproducing content the source never contained. A corrected rate indistinguishable from the floor is a meaningful negative result: it bounds contamination for that model and benchmark, which current methods cannot do without log probabilities.
References
- Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency
- Improved Natural Language Generation via Loss Truncation
- Training Dynamics for Text Summarization Models
- Analyzing and Evaluating Faithfulness in Dialogue Summarization
- On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarization
- Investigating Data Contamination in Modern Benchmarks for Large Language Models
- Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
- Measuring Non-Adversarial Reproduction of Training Data in Large Language Models
- ADP-Hep-Induced Liquid Phase Condensation of TIFA-TRAF6 Activates ALPK1/TIFA-Dependent Innate Immune Responses
- Preservation of ALYREF Phase Separation Mitigates Doxorubicin‐Induced Cardiomyocyte DNA Damage and Cardiotoxicity
- DNA damage-induced paraspeckle formation enhances DNA repair and tumor radioresistance by recruiting ribosomal protein P0
- A Comprehensive Survey of Contamination Detection Methods in Large Language Models
- Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models