Separating Memorized Test Items from Widely Known Facts in Contamination Detection

Tools that check whether a language model saw a benchmark during training mostly ask one question: is the model unusually confident about this exact text? That confidence has two possible causes. The model may have memorized the test item. Or the item may describe something that appears in thousands of independent web pages, so any competent model finds it predictable. Information extraction researchers have relied on the second effect for years. Their premise is that a fact repeated across many independent documents is trustworthy, and they use that repetition to filter errors. Contamination detectors treat the same repetition as a warning sign. We propose to resolve this conflict directly. For each test item, we retrieve naturally occurring, independently written mentions of the same content from an open web corpus. We then ask whether the model's confidence is specific to the item's own wording or spreads evenly across those independent mentions. Confidence concentrated on the item suggests contamination. Confidence that transfers to independent mentions suggests ordinary knowledge.

The practical stakes are that current detectors could be systematically unfair. They may flag benchmarks about well-covered topics as contaminated, and they may clear rare items that leaked verbatim a handful of times. We would test this in a controlled setting where the ground truth is known. We then check whether a detector calibrated against corpus redundancy recovers the performance of an expensive reference model without training one.

Core Claim

Reference-free membership inference scores, such as Min-K% Prob and its normalized successor Min-K%++, are confounded by the item's content redundancy. Content redundancy means how many lexically independent documents in the pretraining distribution express the same facts. Membership inference is the task of deciding whether a text was in a model's training set. We claim that subtracting the model's score on independent natural mentions of the item's content removes most of this confound. Let $s(\cdot)$ be any base detector score and let $N(x)$ be a set of mentions of item $x$'s content retrieved from an open corpus. The mentions are filtered so that none is a near-duplicate of $x$ itself. The calibrated score is

$$C(x) = s(x) - \frac{1}{|N(x)|}\sum_{m \in N(x)} s(m),$$

and it falls back to $s(x)$ when $N(x)$ is empty.

Our specific prediction is this. Among items that were not in training, raw scores rise with content redundancy and $C(x)$ does not. Among items that were in training, $C(x)$ separates them from non-members at every redundancy level, including the rare-fact tail where the item is the model's only source.

The step that makes this more than retrieval is the independence requirement from redundancy-driven extraction. Yangarber et al. showed that corpus consensus corrects extraction errors, turning 68.5% of erroneous records correct with a simple majority vote and 76.4% with confidence ranking. Their premise is that errors are sporadic while true facts recur across sources. The known failure of that premise is shared boilerplate: copies of one source masquerade as consensus. For contamination the same failure is the phenomenon itself. A benchmark mirrored on 200 sites is one source repeated, not 200 witnesses. We therefore count copies of $x$ as item exposure and count only lexically independent documents as content redundancy. A detector that confuses the two is exactly what we expect current methods to be.

Relation to Prior Work

Three kinds of prior work shape the design. Some motivates the gap, some supplies methods we reuse, and some constrains what can work.

The closest work is Magar & Schwartz. They pretrained BERT models with test sets injected at controlled duplication counts. They found that memorization need not lead to exploitation, and that exploitation reaches 40% when a test example appears 200 times. Their redundancy is verbatim duplication of the item. They never vary how widely the item's content is known independently of the item, and they study encoder models on classification rather than detection. We adopt their controlled-injection method but add the second axis, content redundancy, and move to decoder models and detection.

DCR defines a "semantic-level" category of contamination and applies a single penalty factor to accuracy. It relies on static probes and does not try to tell legitimate world knowledge apart from leakage. PAC and CAP compare the model's behaviour on an item against synthetic neighbours: word swaps for PAC, and shuffles and paraphrases for CAP. Our neighbours are instead naturally written documents from the corpus. Our hypothesis is that synthetic neighbours inherit the item's topic and wording and so cannot tell a widely known fact from a leaked one. An ablation tests this.

From the extraction side we borrow two things. The first is the high-precision, low-recall mention-harvesting logic of Fleischman et al.. Strict rules that miss most mentions still recover enough mentions at corpus scale, which is all $N(x)$ needs. The second is the pattern-and-tuple view of Hassan et al., which we use to decide what counts as the "content" of an item.

Two constraints limit the design. Detectors are fragile on short texts: AECA degrades at 32 tokens or fewer. Averaging over external mentions should help precisely there, which gives us a testable side prediction. The second constraint is a real limitation: redundancy methods break down on rare facts. We report every result binned by content redundancy so the tail is visible rather than averaged away.

Research Program

The first component is a controlled testbed. We would pretrain decoder-only models of roughly 150M to 1B parameters on a fixed slice of FineWeb. We then inject a grid of probe items that crosses two factors. The first is content redundancy: the item's underlying facts appear in zero, few, or many independently written documents. We produce these documents by selecting naturally frequent versus rare entity-relation facts already present in the slice, and by planting a controlled number of independently phrased synthetic documents for otherwise novel facts. The second factor is item exposure: the item itself is absent, present once, or duplicated 10 or 100 times, verbatim or as a single paraphrase. This testbed gives exact membership labels and exact redundancy counts, which no public benchmark provides.

The second component is the mention harvester. For each item we identify its content with a lightweight, high-precision extractor. For fact-bearing text these are entity pairs and relation phrases. For math word problems they are entity, quantity, and structure signatures. We then retrieve matching passages from a held-out open corpus that is disjoint from the training slice. We remove near-duplicates of the item itself using MinHash similarity on n-grams above a threshold that we tune once on the testbed, and we cap the number of passages per domain. The number of surviving independent sources is the item's estimated content redundancy.

The third component is the calibrated detector $C(x)$, built on top of Min-K%++. It also works over any base score, so we would apply it to perplexity and Min-K% as well. For closed APIs without log-probabilities, the same subtraction can use PAC's logit-bias probing, at higher cost.

Baselines

The naive reference point is raw sequence perplexity, plus exact 13-gram overlap against the training corpus when that corpus is available. Perplexity is known to be dominated by common tokens. AECA beats it by 4.5% AUC on 256-token WikiMIA sequences (Ke et al.). N-gram overlap is the right "do nothing clever" check wherever the corpus is open.

The strongest reference-free competitor is Min-K%++. It reports AUROC gains of 6.2% to 10.5% over Min-K% on WikiMIA, and parity with reference-model methods on MIMIR (Zhang et al.). The file reports only these relative gains; no reported result found on Lacuna for the absolute WikiMIA AUROC of Min-K%++, so we would rerun it ourselves on identical models and splits. Beating it means a higher AUROC and higher true positive rate at a 5% false positive rate on the redundancy-crossed testbed. The gains should concentrate in the high-redundancy non-member cells and the short-item bin, and WikiMIA should not get worse.

PAC is the strongest perturbation-based competitor. It reports average AUC gains of 4.5% on WikiMIA and 5.9% on StackMIA over prior baselines (Ye et al.). We run it with the same number of neighbour forward passes per item as our method, so the compute budget is matched.

The oracle is a reference-model likelihood ratio. The reference model has the same architecture and is trained on the same slice without the injected items. It controls for world knowledge by construction and is feasible only in our testbed. We treat it as the ceiling we hope to approach, not a competitor we must beat.

The key ablation replaces natural mentions $N(x)$ with an equal number of LLM-generated paraphrases of $x$. This isolates whether independence of the neighbours is what matters. A second ablation removes the near-duplicate filter, so that mirrored copies of the item count as redundancy. If we are right about the boilerplate failure mode, detection of duplicated items should collapse.

Why not just…?

Why not just use n-gram overlap against the corpus? Where the training corpus is open, overlap is a strong detector of verbatim leaks, and we report it as such. It cannot score closed models whose data are undisclosed, which is the main auditing setting. It also misses paraphrased leaks, which CAP shows models still exploit (Zhao et al.). The testbed's paraphrased-exposure cells measure this directly.

Why not just train a reference model? This is the real threat. A clean reference model trained on the same distribution already absorbs world-knowledge redundancy, so a likelihood ratio against it should disentangle the two causes of confidence. For frontier models, however, it requires re-pretraining on an unknown mixture, which Shi et al. identify as infeasible. The contribution is therefore a reference-free approximation. We would count the project successful only if $C(x)$ closes most of the gap between Min-K%++ and the oracle in the testbed. If it closes little, the idea fails.

Why not just use synthetic perturbations as CAP and PAC do? Paraphrases of $x$ share its topic and much of its wording. A model that knows the topic will be confident on both, and a model that memorized $x$ may also transfer some of that confidence to close paraphrases. Neither case produces a clean contrast. Naturally written documents vary everything except the facts. The paraphrase-for-mention ablation settles which view is correct. If synthetic neighbours match natural ones, the extraction component is unnecessary and we would say so.

Proposed Experiments

The first experiment is diagnostic and can fail on its own terms. In the testbed, we plot each baseline's score for non-member items against their harvested content redundancy. If Min-K%++ scores on non-members do not rise with redundancy, the confound we posit is negligible and the project reduces to a negative result.

The second experiment is the main comparison. We report AUROC and true positive rate at 5% false positive rate for all baselines and $C(x)$, broken out by redundancy bin, exposure count, verbatim versus paraphrased exposure, and item length. We also run WikiMIA and MIMIR, giving special attention to WikiMIA's 32-token split.

The third experiment is an external check with no controlled labels. Zhang et al. found that a model's GSM8K generation log-likelihood predicts its accuracy drop on the held-out GSM1k with $r^2 = 0.36$, across model families that fell by up to 8%. We compute model-level mean $C(x)$ over GSM8K for the same open models. The decisive check is whether this mean predicts the GSM8K-to-GSM1k gap better than raw log-likelihood does. It should, because widely repeated problem patterns inflate the raw score regardless of leakage.

The fourth experiment examines where the method breaks. On tail items with empty $N(x)$, the method reduces to the base detector by construction. We report how many items in each public benchmark fall in that bin, so that auditors know where the calibration applies and where it does not.