When a benchmark question leaks into a language model's training data, people usually check for it in one of two ways. The first looks for long verbatim word overlaps. The second compares sentence meanings with a neural embedding. Both fail in predictable directions. Verbatim matching misses rewritten copies: an LLM-paraphrased benchmark slips past it almost entirely. Meaning-based matching flags too much: on templated benchmarks such as grade-school math, thousands of different problems about apples and trains look alike, so every candidate has to be checked by an expensive LLM judge. This proposal starts from a simple observation. A rewrite that preserves a test item's answer must also preserve its anchors: the specific numbers, named entities, answer options, formulas, and code identifiers that fix what the item is asking. We propose a two-channel contamination search. A dense channel finds candidates with the same meaning even when the narrative is rewritten. A character-level lexical channel then checks whether the candidate carries the item's anchors, tolerating formatting variation such as "1,200" versus "1200", x_1 versus x1, or LaTeX versus Unicode fractions.
The payoff is a contamination audit that stays cheap at web-corpus scale and still catches paraphrased leakage. It also yields a new kind of measurement. Every corpus passage gets two coordinates relative to a test item: how close it is in meaning and how many anchors it shares. That two-dimensional map separates instance duplicates (same problem, reworded) from template duplicates (same kind of problem, different specifics). We can then test directly, with controlled fine-tuning, which of the two actually inflates benchmark scores.
Core Claim
Our central claim is that anchor agreement, measured with character n-grams, can stand in for most of the LLM-judge verification that dense contamination search currently needs. Character n-grams here are overlapping character substrings of length 3 to 6, the unit FastText uses to represent words it has never seen. Concretely, for a test item $q$ and a corpus span $d$, we score
$$s(q,d) = \cos\big(e(q), e(d)\big) \quad\text{and}\quad a(q,d) = \frac{1}{|A_q|}\sum_{x \in A_q} \max_{y \in d} J_3(x, y),$$
where $e(\cdot)$ is a dense sentence encoder, $A_q$ is the set of anchor strings extracted from $q$, and $J_3(x,y)$ is the Jaccard overlap between the character 3-to-6-gram sets of anchor $x$ and a window $y$ of the span. High $s$ with high $a$ marks an instance duplicate. High $s$ with low $a$ marks a template duplicate. The claim is testable. On controlled injections, the pair $(s,a)$ should reach a detection rate close to dense search plus an LLM judge, at a small fraction of the cost. Against verbatim filters it should do far better on paraphrased items. A secondary, scientific claim is that the $(s,a)$ quadrants differ in how much they inflate scores.
Relation to Prior Work
Two papers motivate the gap. Evading Data Contamination Detection for Language Models is (too) Easy shows that GPT-4 paraphrasing of benchmark items, which they call Evasive Augmentation Learning, drives sample-level detection to about 1% true positive rate at a 1% false positive rate while inflating GSM8K, MMLU, ARC-Challenge and TruthfulQA scores by up to 15%. Soft Contamination Means Benchmarks Test Shallow Generalization searches the Olmo3 training data with dense cosine similarity followed by LLM verification. It finds semantic duplicates for 78% of CodeForces tasks and shows that fine-tuning on semantic duplicates lifts MuSR by about 20%, without transfer to related benchmarks. That paper is our closest neighbor. We keep its dense stage but replace most of its LLM verification with a cheap lexical check. It treats "semantic duplicate" as one category; we split it into instance and template duplicates and measure the two separately.
Two papers constrain the design. Does Data Contamination Detection Work (Well) for LLMs? reports that probability-based detectors sit near 0.50 AUC on pretraining data for Pythia and OLMo-2, and work only after fine-tuning-style repetition (99.4 AUC on RE-2012temp). This is why we search the corpus rather than probe the model. Search-Time Data Contamination shows that web search agents pull answer keys from benchmark mirrors. We treat that as a secondary application, because the same anchor check can run on retrieved context before it reaches the model.
The method comes from hybrid retrieval and subword representations. Sparse, Dense, and Attentional Representations for Text Retrieval shows that interpolating sparse and dense scores recovers exact rare strings that dense encoders blur. It also argues that fixed-length dense vectors lose fidelity on long documents. That argument is why we embed short sliding windows rather than whole web pages, since leaked items are typically short spans inside long pages. Hybrid First-stage Retrieval Models for Biomedical Literature casts BM25 as a sparse dot product concatenated with dense vectors. It uses word bigrams and BERT pieces; we use character n-grams restricted to anchor strings, which keeps the sparse index small. Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval supplies the dimensionality reduction (PCA plus product quantization) that makes the dense stage affordable over a FineWeb sample. It also warns that compression hurts out-of-domain retrieval, which we monitor. A Systematic Study of Leveraging Subword Information and What to do about non-standard language in NLP justify character n-grams as stable under surface variation. The first finds that simple additive composition is robust; the second finds that character models degrade less as the out-of-vocabulary rate rises. The same subword study also warns that n-grams collide spuriously. In our setting, that warning suggests requiring several anchors to agree rather than trusting any single one.
Research Program
The work has three parts.
The first part builds the anchor extractor and the index. Anchors are extracted by rules that we validate by hand. The rules capture numbers with their units, quoted entities and proper nouns, multiple-choice option strings, mathematical expressions, and identifiers and literals in code. Each anchor is normalized: whitespace and case are stripped, and digits and LaTeX are canonicalized. It is then indexed as a bag of character 3-to-6-grams. Corpus documents are cut into overlapping windows of about 128 tokens. Each window is embedded with a compressed dense encoder, and its anchor-like strings are indexed in an inverted index.
The second part builds the benchmark for contamination detection. We take GSM8K, MMLU, ARC-Challenge, MBPP, ZebraLogic and CodeForces items and generate five variants of each: verbatim; EAL-style LLM paraphrase; an ITD-style narrative rewrite that keeps the arithmetic, following Inference-Time Decontamination; a template variant whose numbers or entities are changed; and an adversarial formatting perturbation of the anchors. We inject these variants as spans inside random long documents from a FineWeb sample (FineWeb, resource). Because we know where every variant sits, ground truth is exact.
The third part is a causal study. We fine-tune a small open model from the Pythia or OLMo family on items drawn from each $(s,a)$ quadrant at matched contamination rates. These include the very low rate (4 in 10,000) at which Soft Contamination found a 15% gain. We then measure score inflation on the seen items and on held-out items from the same benchmark. Finally, we audit the real Olmo3 training data and compare our quadrant counts against Soft Contamination's reported prevalence.
Baselines
The naive reference point is the industry-standard verbatim filter: 13-gram exact overlap, plus MinHash near-duplicate detection as used in FineWeb-style pipelines. This is what decontamination does today. Against paraphrased items, the EAL paper reports it at about 1% TPR at 1% FPR. We run it on the identical injected corpus and score it with the same metric. Any detector that cannot beat it on the paraphrase variants is uninteresting.
The strongest existing method is the Soft Contamination pipeline: dense cosine retrieval followed by LLM-judge verification. It is the only cited method designed for semantic duplicates in a real pretraining corpus. On the Olmo3 data it reports 78% semantic-duplicate prevalence for CodeForces and 50% exact duplicates for ZebraLogic. We found no reported result on Lacuna for its detection rate on controlled paraphrase injections, so we will measure that rate ourselves. We run it with the same encoder, the same windows, the same top-$k$ candidates and the same judge model as our system. We also report it without the judge, as pure dense thresholding. Beating it means one of two outcomes. Either we match its TPR at 1% FPR to within a few points while using under a tenth of its judge calls, or we exceed it outright on the template-versus-instance distinction, scored against our injection labels.
The standard hybrid competitor is linear interpolation of word-level BM25 and dense scores in the ME-HYBRID style from Sparse, Dense, and Attentional Representations for Text Retrieval. That paper reports 0.343 MRR@10 on MS MARCO Passage, against 0.302 for a BERT dual encoder. We give its interpolation weight the same tuning budget, on a held-out validation split of injections.
Two ablations isolate our ingredients. The first swaps character n-grams for word or BPE tokens over the same anchors. If this matches the full system, the character component is unnecessary and we will say so. The second applies character n-grams to the whole window rather than only to anchors, which tests whether restricting the check to anchors is what buys precision. MinKProb is included as the model-probing reference, on the Pythia checkpoints where pretraining membership is known.
Why not just…?
Why not just use dense retrieval with an LLM judge, as Soft Contamination does? This is the real threat, and it may match us on raw detection rate. Our argument is cost and resolution, not accuracy alone. A judge call for every dense candidate is the expensive step that limits audits to benchmark-sized query sets, and the judge returns one undifferentiated "duplicate" label. The experiments settle the question in two ways. We report agreement between the anchor score and judge verdicts, and detection rate as a function of judge calls spent. If the judge remains necessary even for high-$a$ candidates, the contribution shrinks to the quadrant analysis, and we would report it that way.
Why not just use 13-gram matching or MinHash? The EAL paper already shows these collapse to about 1% TPR against paraphrase. Our injection benchmark reproduces that check, and adds the narrative-rewrite variants, where anchors survive but word sequences do not.
Why not probe the model with perplexity or MinKProb and skip corpus search? Does Data Contamination Detection Work (Well) for LLMs? finds these at about 0.50 AUC on pretraining data, because a model that is competent in a domain assigns low loss to all of that domain's text. We include MinKProb on Pythia only as a reference point.
Why not use word-level BM25 on anchors instead of character n-grams? This is plausible. Most anchors are numbers and names that word tokenizers keep intact. Character n-grams should matter for formatting perturbations, LaTeX, and code. The token-level ablation answers the question directly, broken down by perturbation type. If the gain appears only under adversarial formatting, we will claim robustness to that attack and nothing broader.
Decisive Checks
The main result is TPR at 1% FPR for each variant type, on spans injected into FineWeb windows, alongside judge calls per detected item. The proposal succeeds if three things hold. Anchor-gated hybrid search beats 13-gram matching by a wide margin on paraphrase and narrative rewrites. It stays within a few points of dense search plus judge at a fraction of the judge calls. And it beats word-level anchors on formatting perturbations. The scientific check is the fine-tuning study. If instance duplicates, meaning high $a$, inflate seen items much more than template duplicates do, while template duplicates drive the held-out gains Soft Contamination observed, then the two-axis map measures something that a single similarity score cannot. If the quadrants behave identically, the map has no scientific value, and the contribution reduces to a cheaper detector.
References
- Evading Data Contamination Detection for Language Models is (too) Easy
- Soft Contamination Means Benchmarks Test Shallow Generalization
- Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptions
- Search-Time Data Contamination
- Sparse, Dense, and Attentional Representations for Text Retrieval
- Hybrid First-stage Retrieval Models for Biomedical Literature
- Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval
- A Systematic Study of Leveraging Subword Information for Learning Word Representations
- What to do about non-standard (or non-canonical) language in NLP
- Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale