Finding Paraphrased Benchmark Leaks by How Tightly Their Words Cluster

When a benchmark question leaks onto the web, it rarely stays verbatim. Instead it shows up reworded inside a forum post, a solutions blog, or a scraped homework page. Almost always, though, its rare words (names, numbers, variable identifiers, unusual nouns) sit packed together in one short stretch of an otherwise unrelated long page. The two tools used today to audit pretraining corpora both miss this. Exact 13-word overlap breaks as soon as a scraper rewords the question. Dense embedding similarity, which compares learned vector summaries of whole texts, is expensive at web scale and blurs a short leaked passage into the topic of the surrounding page. This proposal borrows an old idea from search engines, term proximity scoring: rank a document by how close together the query's words appear, not just whether they appear. The proposal turns that idea around and uses it as a contamination audit. The question becomes: does a short window of this web page contain most of the rare vocabulary of this benchmark item?

The claim is that a proximity score computed over a positional inverted index can recover most paraphrased benchmark leaks that embedding search recovers, at a small fraction of the compute, and that it clearly beats exact n-gram matching and whole-document bag-of-words retrieval. A second claim is that the items it flags are the ones where models score suspiciously well. If both hold, auditing a 15-trillion-token corpus such as FineWeb for paraphrased leaks becomes a routine indexing job rather than a dense-retrieval project.

Why this gap exists

The contamination literature has documented the problem well but has inherited its detection tools from two extremes. A Comprehensive Survey of Contamination Detection Methods in Large Language Models surveys open-corpus auditing by exact string matching, embedding similarity, and LLM-based decontaminators, and notes that string matching is defeated by rewording. Soft Contamination Means Benchmarks Test Shallow Generalization motivates this work most directly. It finds semantic duplicates of 78% of CodeForces problems and exact duplicates of 50% of ZebraLogic in Olmo3 training data, and shows that even a contamination rate of 4 in 10,000 can inflate scores by around 15%. Its method, however, relies on embedding similarity followed by LLM verification, which it describes as both costly and prone to underestimating leaks. Model-side detectors take a different approach. MIN-K% PROB averages the log-probabilities of a text's least likely tokens, and Min-K%++ normalizes those log-probabilities against the model's full output distribution. Both work without the corpus, but they need token probabilities and they say nothing about where a leak lives, so they cannot be used to clean data.

On the retrieval side, structured engines such as Indri operationalized proximity as ordered windows (exact phrases) and unordered windows (all terms within N tokens, in any order). ICTNET at Web Track 2010 instead used the smallest span of text containing all query terms. These are the methods this proposal reuses. Adapting Learned Sparse Retrieval for Long Documents constrains the design in a useful way. It shows that in long documents, summing or maxing scores across segments lets irrelevant text dilute the signal, while an exact-term proximity model (ExactSDM, 8-token windows) fixes this. A variant that also matches neural expansion terms (SoftSDM) adds nothing. That last result is exactly the one this proposal must re-test, because in contamination the whole difficulty is rewording, which is where soft matching ought to matter. Search-Time Data Contamination shows the same leaks reaching search agents at inference time. It treats the search engine as a black box, so the mechanism studied here is relevant to it, but it is not the focus.

No paper in the file applies proximity scoring to contamination auditing. The closest work, the soft-contamination study, finds paraphrased leaks with dense embeddings. This proposal replaces that step with a lexical, position-aware score and asks how much recall is kept per unit of compute.

Core Claim

For a benchmark item $q$ with content terms $T_q$ (stopwords removed), each weighted by its corpus rarity $w(t)$ (inverse document frequency), and a document $d$, define the leak score as the largest rarity-weighted fraction of the item's vocabulary found inside any window of $d$ no longer than $\alpha$ times the item's length $|q|$:

$$S(q,d) = \max_{W \subset d,\; |W| \le \alpha |q|} \frac{\sum_{t \in T_q \cap W} w(t)}{\sum_{t \in T_q} w(t)}.$$

Here $W$ ranges over contiguous token spans of $d$, and $\alpha$ (expected around 1.5 to 3) allows for paraphrases that are somewhat longer than the original. This is the minimum-window idea, reweighted by rarity and normalized by item length. The bet behind it is that paraphrasing changes function words, word order, and common verbs, but rarely changes the rare tokens that make a question that particular question. Taking the maximum over windows prevents the surrounding page from diluting the score, which is the long-document failure identified in the learned-sparse-retrieval paper. A secondary feature, the fraction of matched terms that appear in their original order, separates verbatim copies from paraphrases for reporting purposes.

Research Program

The system has three stages, each sized for a trillion-token corpus. The first stage builds a positional index restricted to the union of benchmark content vocabulary. Because benchmark items contain rare tokens, these position lists cover a small part of the corpus, and storing positions only for these terms avoids most of the overhead that makes full positional indexes costly. The second stage generates candidates: any document holding at least a fraction $\beta$ of an item's rare-term mass becomes a candidate, a bag-of-words filter computable from postings alone. The third stage computes $S(q,d)$ exactly over the candidate's positions with a sliding window and passes items above a threshold to a fixed LLM verifier. The verifier judges whether the window states the same task, and the same verifier and prompt are used for every method compared.

The analysis has three threads. The first measures how recall depends on window size $\alpha$ and on paraphrase strength, to find where proximity stops helping. The second re-tests the ExactSDM-versus-SoftSDM question in this new setting by adding a light expansion step, so that a matched term can also be a lexical variant or a SPLADE expansion term. If expansion helps here when it did not help for ad-hoc retrieval, that explains the tension between the two literatures. The third checks validity downstream: whether flagged items behave like contaminated items in model scores.

Baselines

The naive reference point is 13-gram exact overlap, the standard open-corpus check described in the survey. It runs over the same corpus sample and the same benchmark items, and any document sharing a 13-gram with an item counts as flagged. This baseline sets the floor for verbatim leaks. The proposal should match it on verbatim cases and exceed it substantially on paraphrased ones.

The strongest existing method is embedding retrieval with LLM verification, as in Soft Contamination Means Benchmarks Test Shallow Generalization. Corpus chunks are embedded, the nearest neighbors of each benchmark item are retrieved, and the same verifier screens the results. This is the method that actually found the large paraphrased leak rates. The file reports prevalence figures (78% of CodeForces as semantic duplicates, 50% of ZebraLogic as exact duplicates in Olmo3 data), but there is no reported result on Lacuna for this method's detection recall or precision against labeled ground truth. The comparison will therefore produce that number rather than quote one. For fairness, both methods get the same corpus sample, chunking at the same granularity used for verification, and the same verifier call budget per item. Tuning is limited to the same small grid: chunk size and $k$ for embeddings, $\alpha$ and $\beta$ for proximity. Cost is reported in CPU and GPU hours. Proximity "beats" this baseline if it reaches at least 90% of its verified recall on paraphrased leaks at a tenth of the compute or less, or exceeds its recall at equal compute.

A third baseline is whole-document BM25 retrieval feeding the same verifier. This is bag-of-words ranking with no positions, and it is the cheapest competitor that tolerates rewording. The key ablation removes the window from $S$, so the score is computed over the entire document. This turns $S$ into rarity-weighted bag-of-words coverage and isolates exactly what proximity contributes. If the windowed score does not beat this ablation on leaks embedded in long pages, the central idea is wrong. Further ablations remove the rarity weights and add soft expansion, as described above.

Finally, Min-K%++ is a model-side reference point on items for which an open model's training data is known. It is not a competitor for cleaning a corpus, but it lets us test whether corpus-side flags and model-side membership scores agree.

Why not just…?

Why not just use 13-gram matching, which is cheap and already standard? Because it measures the wrong thing for the leaks that matter. The soft-contamination study found that exact duplicates cover only part of the problem, with semantic duplicates dominating CodeForces, and the survey documents how rewording evades n-gram checks. The controlled injection experiment below measures this directly at each paraphrase strength. If 13-grams recover most paraphrased injections, the proposal is unnecessary, and that outcome would be reported.

Why not just run dense embedding search, now that approximate nearest-neighbor indexes are fast? This is the real threat. Embedding search may well have higher recall on heavy rewording, where even rare terms are swapped out. The proposal's case rests on cost and on long-document dilution: embedding a trillion tokens in chunks is a large GPU job, and a short leak inside a long chunk can be hidden by the chunk's main topic. The head-to-head comparison at equal verifier budget settles this. If embeddings dominate even after accounting for cost, the honest conclusion is that proximity is a cheap pre-filter at best, and we would report the combination instead.

Why not just use BM25 plus a verifier, skipping positions entirely? This tests whether proximity adds anything beyond rare-term overlap. The whole-document ablation of $S$ is designed to answer it. The prediction is that the advantage appears specifically for leaks inside long documents, mirroring the long-document result in Adapting Learned Sparse Retrieval for Long Documents. If there is no advantage there, positions are not worth their storage.

Why not just use Min-K%++ on the model and skip the corpus? It needs log-probabilities. It reports performance in relative terms, around a 6.2% to 10.5% AUROC gain over Min-K% on WikiMIA. And it cannot locate or remove a leak. Corpus auditing answers a different question. The two are compared only to test agreement.

Proposed Experiments

The first experiment is controlled injection. A FineWeb sample (resource) on the order of tens of billions of tokens is seeded with items from GSM8K, MMLU, ARC-Challenge, and MBPP at five paraphrase strengths: verbatim, reordered, synonym-substituted, LLM-rewritten, and rewritten with renamed entities and numbers. Each item is inserted at a random position inside a long unrelated host document. The metrics are recall at fixed precision after verification, and recall per compute hour, for every method and ablation. This experiment shows where the rare-term assumption breaks: renamed numbers should hurt proximity most, and if they do, that defines the method's honest limit.

The second experiment is natural-leak recovery on the Olmo3 training data analyzed in Soft Contamination Means Benchmarks Test Shallow Generalization. The audit is run on CodeForces, ZebraLogic, MuSR, and MBPP, and the verified flags are compared with the reported prevalences (78% semantic duplicates for CodeForces, 50% exact duplicates for ZebraLogic). Where the two methods disagree, a sample of disagreements is labeled by hand to see which method is right.

The decisive check is downstream validity. For open models whose pretraining corpus is FineWeb-derived or Olmo3, accuracy on proximity-flagged items is compared against accuracy on unflagged items of similar difficulty. Difficulty is matched using clean reference models in the manner of ConStat, which reports gaps such as about 8.3% for Mistral-7b on GSM8K. The same comparison is made for items flagged only by proximity versus only by embeddings. The proximity audit earns its place if its flagged items show an inflation gap at least as large as the embedding-flagged items, at a fraction of the compute. If its flagged items show no gap, its flags are lexical coincidences rather than leaks, and the proposal fails.