Contamination detectors for large language models answer a yes-or-no question. Given one test item, they guess whether the model saw it during training; given a whole benchmark, they test whether any leakage exists. What evaluators usually need is a number: roughly how many of these 1,000 questions did the model see, and how much would its score fall without them? That is a recall problem in disguise. Every detector misses some leaked items, and nobody has a list of all leaked items to check against. This is the same missing-denominator problem that information-extraction researchers face when they try to measure recall over the open web. Ecologists face it too when they count fish in a lake. Their standard answer is capture–recapture: catch animals twice with independent methods, and estimate the total population from how many animals were caught both times.
We propose to treat each contamination detector as a separate "capture" of leaked benchmark items. The total leaked population is then estimated from how much those captures overlap. The detectors must rest on mechanistically different signals: token likelihood, consistency under perturbation, and verbatim continuation. The ingredient that makes this practical is an asymmetry in what is easy to label. Items guaranteed to be clean are cheap to obtain, for example questions published after a model's training cutoff. Items guaranteed to be leaked are not. Clean items give each detector's false-positive rate directly. The overlap structure then supplies each detector's sensitivity, which no existing method estimates without labeled positives. The output is an estimated count of contaminated items with a confidence interval, plus a contamination-adjusted benchmark score.
Core Claim
The claim is that the number of leaked items in a benchmark can be estimated without any positive labels, and with honest uncertainty. It cannot be done with any single detector, however good. It can be done by combining several detectors whose errors are only weakly correlated, using population-size estimators that model the dependence between detectors. In its simplest form, with two detectors that flag $n_1$ and $n_2$ items, $m$ of them flagged by both, the Lincoln–Petersen estimate of the total leaked count is
$$\hat N = \frac{n_1 n_2}{m},$$
after each $n_i$ and $m$ have been corrected for the false-positive rate measured on clean items. The formula holds when the two detectors catch leaked items independently. Real detectors violate this, because items that are memorized strongly are easier for every detector to catch. We therefore use three or more captures, combined with log-linear models of the kind used in multi-list population estimation. These models fit pairwise dependence terms and treat heterogeneity in how "catchable" each item is as an explicit, testable quantity rather than a hidden bias.
Relation to Prior Work
The motivation comes from two separate literatures. On the recall side, Importance sampling for unbiased on-demand evaluation of knowledge base population shows that the standard practice of pooling outputs from several systems and judging only the pool biases recall. Scoring unpooled outputs against that pool carries a median macro-F1 bias of 15.51%. Its importance-sampling estimator cuts this to 2.05%. That paper still needs human judgments of sampled documents to estimate recall, and it never considers memorization. Completeness, Recall, and Negation in Open-World Knowledge Bases: A Survey catalogues mark-and-recapture as a way to estimate how many facts a knowledge base is missing. It explicitly leaves benchmark leakage unaddressed. We take its estimator family and apply it to a population nobody has applied it to: leaked test items.
On the contamination side, our proposal reuses existing detectors as capture devices rather than competing with them. Those detectors are Min-K%++, a likelihood-based detector that normalizes each token's log-probability against the vocabulary distribution, and the consistency-under-perturbation idea of CAP. PaCoST is the clearest statement of the gap. It runs a paired significance test on confidence for original versus rephrased questions and says plainly that it works only at the level of whole datasets and cannot enumerate leaked items.
The closest competitor is How Contaminated Is Your Benchmark?. Its Kernel Divergence Score (KDS) fine-tunes the model briefly and measures how much its embeddings reorganize. It reaches a Spearman correlation of 0.99 with true contamination proportions. A rank correlation, however, says that KDS orders benchmarks correctly. It does not show that KDS yields a calibrated absolute fraction with an interval, and KDS needs the ability to fine-tune the model. Importance Sampling is All You Need predicts code-benchmark scores from prompt distributions and flags contamination when observed scores diverge from predicted ones. It treats that divergence as an alarm, not as a measured volume.
Recall Residualisation is the nearest in spirit. It regresses a model's forecasting signal on its directly recalled values and reports LeakShare, the fraction of apparent skill explained by memory, which was at least 0.97 in 19 of 20 cases. Its method depends on series that can be addressed by date. Our method needs no such indexing and works over arbitrary question sets.
One paper constrains the design. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that a single epoch of GRPO, a reinforcement-learning method for reasoning models, after contaminated supervised training reduces likelihood detectors to near-random (50–58% AUROC). If every capture is near-random, capture–recapture cannot recover anything. We do not claim to defeat this result. We treat it as the regime where the method's own diagnostics must report failure, and we test whether behavioral captures survive it.
Research Program
There are three components. The first is the capture devices. They are a likelihood capture (Min-K%++ score), a perturbation capture (accuracy and confidence drop between the original item and paraphrased or option-shuffled versions, following CAP), and a continuation capture (prompt with the first half of the item and check whether the model reproduces the remainder or the gold answer verbatim). The continuation capture needs only generated text, so it applies to closed APIs. For open models we add KDS as a fourth capture where fine-tuning is possible.
The second component is calibration on negatives. For each model, we assemble a pool of items that cannot have been seen. These include WikiMIA's post-2023 non-members, and WMDP for models trained before May 2024. For each capture we set the threshold at a fixed false-positive rate on this pool, and we use the measured rate to correct the flagged counts.
The third component is estimation. We fit log-linear models over the 2^k capture histories, select dependence terms with information criteria, and bootstrap over items to obtain intervals. The estimated leaked count then yields an adjusted score. We reweight accuracy on items no detector flagged, using each item's estimated probability of being leaked but uncaught. The result is an estimate of performance on the clean subpopulation.
Baselines
The naive reference point is classify-and-count. One run applies a single detector, Min-K%++, at a threshold tuned on WikiMIA and counts flagged items. A second run uses a perplexity threshold. Both use the same clean-negative pool for thresholding as our method.
The strongest prevalence competitor is adjusted classify-and-count. It corrects a detector's flag rate using a true-positive rate measured on a benchmark that does contain known positives, here WikiMIA members, and transfers that rate to the target. It uses the same detectors and the same negatives as our method. It differs only in where sensitivity comes from: borrowed from a different benchmark, versus estimated from the target's own capture overlap.
The strongest existing contamination-quantity method is KDS. It is run with the same LoRA budget its authors use, on the same models and contamination mixtures. We convert its scores to fractions by fitting a calibration curve on held-out mixtures, which gives it a calibration advantage our method does not have. For KDS, the file reports a Spearman correlation of 0.99 with true contamination proportions. It reports no absolute-fraction error, so no reported result was found on Lacuna for KDS absolute estimation error. For the post-reinforcement-learning regime, the reported number to beat is LiRA's 89.1% AUROC after supervised contamination. After GRPO, all ten detectors averaged only 50–58% AUROC (Fragility).
The key ablation replaces our cross-family captures with three likelihood-family captures: Min-K%, Min-K%++, and loss. This isolates signal diversity as the ingredient that matters. We count our method as beating a baseline if it has lower mean absolute error in the estimated leaked fraction and achieves nominal 90% interval coverage, where the baselines' implied intervals do not.
Why not just…?
Why not just use a calibrated single detector and correct its count? This is the real threat. If a detector's sensitivity were stable across benchmarks and models, adjusted classify-and-count would suffice and would be simpler. The literature suggests it is not stable. Min-K%++ reports that likelihood detectors behave differently on rare or technical sequences. The Fragility paper shows sensitivity changing sharply with the kind of post-training. Our experiments settle the question directly. We transfer the true-positive rate from WikiMIA to MMLU and ARC mixtures and measure the resulting bias. If transferred sensitivity is accurate to within a few points, we will report that the simpler method wins.
Why not just compare accuracy on pre-cutoff versus post-cutoff items? The gap mixes contamination with differences in difficulty and topic between old and new items. The EvoEval drop of 39.4% relative to HumanEval, reported in the contamination survey, cannot be cleanly split between the two causes. The comparison also says nothing about how many items in the old set leaked. We use the clean post-cutoff items only for false-positive calibration, which does not require them to match the old items in difficulty.
Why not just use KDS? KDS needs fine-tuning access, which closed models do not offer. It is also validated as a ranking. Whether its calibrated output gives accurate absolute fractions and honest intervals is exactly what our head-to-head comparison measures. KDS also enters our method as one capture, so if it is strong, the combined estimate inherits that strength.
Proposed Experiments
The ground-truth experiments use controlled contamination. We continue pretraining open models on fractions of MMLU, ARC, HellaSwag and WinoGrande items, varying the leaked fraction from 0 to 45%. That range matches the 1–45% rates the survey reports for open models. We also vary contamination level between data-level (questions only) and label-level (questions with answers), and vary the number of repetitions, so that items differ in how catchable they are. In a reasoning arm, we follow the Fragility protocol on AIME, AMC, OlympiadBench and GPQA Diamond: contaminated supervised fine-tuning followed by one epoch of GRPO.
The decisive checks are three. The first is whether the estimated leaked fraction tracks the injected fraction with low absolute error, and whether 90% intervals cover the truth about 90% of the time under realistic dependence between captures. The second is whether the cross-family method clearly outperforms the same-family ablation. If it does not, signal diversity is not doing the work. The third is whether, after GRPO, the method either still recovers the count through its behavioral captures or widens its intervals and flags its dependence diagnostics instead of returning a confident wrong number. Finally, on the FinEval models where CAP confirmed a 21-question overlap, we check whether the black-box variant's interval contains 21.
References
- Importance sampling for unbiased on-demand evaluation of knowledge base population
- Completeness, Recall, and Negation in Open-world Knowledge Bases: A Survey
- Min-K%++: Improved Baseline for Pre-Training Data Detection from Large Language Models
- CAP: Data Contamination Detection via Consistency Amplification
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- How Contaminated Is Your Benchmark? Measuring Dataset Leakage in Large Language Models with Kernel Divergence
- Importance Sampling is All You Need: Predict LLM's performance on new benchmark by reusing existing benchmark
- Recall Residualisation: Decontaminating Foundation-Model Evaluation on Public Time-Series Benchmarks
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Benchmark Data Contamination of Large Language Models: A Survey