Finding Rephrased Benchmark Leaks with a Compressed Exact Index

When a language model has seen benchmark test questions during training, its scores overstate its ability. To check for this, auditors search the training corpus for the test items. The largest audit so far, Infini-gram mini, uses a compressed full-text index: a structure that stores the text in less space than the original and still answers "where does this exact string occur?" With it, the authors searched 83TB of web text. They found that 74.2% of GSM8K appears in a July 2025 crawl and that 27.7% of MMLU entries are dirty in DCLM-baseline. Their index only finds exact matches, though. Yang et al. showed that rephrased test items slip past exact matching while still raising a 13B model from about 55 to about 86 on MMLU. Their fix, which retrieves look-alike passages with a dense embedding index and has GPT-4 judge them, has only been run on corpora far smaller than a web crawl. The audits that run at scale are blind to paraphrase, and the audits that catch paraphrase do not run at scale.

We propose to close most of this gap without a vector index. The corpus is indexed a second time, in a canonical form that erases surface differences such as case, punctuation, whitespace and specific numbers. Each benchmark item is then turned into a handful of short anchor phrases, chosen because the index's own occurrence counts say they are rare. Any document containing several of these anchors goes to an LLM judge. The search keeps the memory profile of the exact index, roughly 0.44× corpus size on disk and tens of megabytes of RAM per query. The claim to be tested is that, at matched storage, this catches nearly all the rephrased contamination that a dense retriever catches, except for translation. We will measure that exception rather than hide it.

Core Claim

Most rephrasings that inflate scores keep a few rare surface fragments of the original item: entity names, unusual noun phrases, code identifiers, and the skeleton of a word problem with its numbers swapped. A compressed exact index over a canonical projection of the corpus can find those fragments cheaply. The index can also count how often any candidate fragment occurs, so it can pick fragments that are rare enough to be evidence of a leak instead of chance overlap.

The selection rule is simple. Let $c(s)$ be the canonical-corpus count of a span $s$ drawn from item $x$. We choose a set $A(x)$ of non-overlapping spans with $c(s) \le \tau$. Then we flag a document $d$ as a candidate when it contains at least $m$ anchors from $A(x)$. Under an independence approximation, the expected number of chance candidates is about

$$ \mathbb{E}[\text{false candidates}] \approx \binom{|A(x)|}{m}\, \prod_{s} \frac{c(s)}{D}, $$

where $D$ is the number of documents and the product runs over any $m$ anchors. This comes from treating anchor occurrences as independent random placements. The approximation tells us how to set $\tau$ and $m$ so that the judge sees a bounded number of candidates per item.

What is new is not the FM-index itself. It is three things together: the canonical projection, anchor selection driven by the index's counts, and a direct test of whether this matches dense retrieval at equal storage on rephrased leaks.

Research Program

The index. We use the FM-index engine from Infini-gram mini. An FM-index is built from the Burrows-Wheeler transform, a reversible reordering of the text, combined with compact bit structures that count symbols quickly. We build it twice over the same corpus:

  • over the raw bytes, as in the original work;
  • over a canonical projection: Unicode-normalized, lowercased, punctuation and whitespace collapsed, every digit run replaced by one placeholder symbol, and common LaTeX and markdown math markup flattened.

The projection loses information. We therefore store a sparse sampled map from canonical offsets back to raw offsets, in the spirit of the sampled positions FM-indexes already keep. A canonical hit can then be shown to the judge as raw text. Raw-text reconstruction from a compressed index is slow, which the file notes as a known limitation. The design only reconstructs a bounded window around each candidate.

Anchors. For each benchmark item, we enumerate canonical spans of 4 to 12 tokens and query their counts. Count queries are the cheap operation in an FM-index. We greedily keep the rarest non-overlapping spans that fall under $\tau$. For GSM8K-style items, number masking turns "changed the numbers" variants into exact canonical matches. For code, identifiers and docstring phrases act as anchors.

Verification. Documents containing at least $m$ anchors are scored by a judge using the semantic-equivalence prompt of Yang et al., run with an open-weight model so that cost is controlled and the results can be reproduced. We record whether the candidate holds the question alone, or the question and its answer. These correspond to the data-level and label-level contamination of the survey's taxonomy.

Planted ground truth. No corpus comes with labels for rephrased leaks, so we build them. We insert known contaminants into a held-out slice of corpus before indexing. These come in five types:

  • verbatim copies;
  • LLM paraphrases in the style of Yang et al.;
  • number-perturbed rewrites of GSM8K items;
  • chain-of-thought solutions written around the question, the contamination type that the fragility study shows defeats probability-based detectors;
  • translations into French and Chinese.

Planting each type at known rates lets us measure recall per type.

Baselines

The do-nothing reference point is the raw-text Infini-gram mini index queried with whole benchmark items, which is the audit as published. Every reported dirty rate, such as 27.7% of MMLU in DCLM-baseline, comes from this procedure. Running it on the same corpus slice tells us how much contamination the extra machinery adds. A second naive reference is 13-gram overlap filtering, the common decontamination practice. It runs as a hash join over the same slice.

The strongest competitor is the dense-retrieval-plus-judge pipeline of Yang et al., which reports a detection F1 of 0.94–1.0 on its rephrased-sample settings. Those settings are their own (HumanEval and MMLU against StarCoder, RedPajama and CodeAlpaca), so the figure is a ceiling to approach, not a number measured on our corpus. We will run it twice:

  • uncompressed, on a slice small enough that this is feasible;
  • as a quantized inverted-file vector index whose disk footprint is matched to our two FM-indexes combined, about 0.88× text.

For the matched-footprint variant there is no reported result found on Lacuna for compressed dense retrieval used in contamination detection, so we will establish it. Both dense variants use the same chunking, judge model and judge budget as our method, with top-$K$ chosen so that the judge sees the same number of candidates per item. We beat the baseline if, at matched footprint and judge budget, we reach recall within a few points of uncompressed dense retrieval on paraphrase, number-perturbed and chain-of-thought plants, and exceed the matched-footprint dense index. We concede translation in advance.

Two ablations isolate the key ingredients. The first queries count-selected anchors against the raw index, with no canonical projection. This tests whether normalization matters or whether anchors alone do the work. The second keeps the canonical index but selects anchors at random among spans of the same length. This tests whether count-based rarity selection is what keeps candidate sets small.

Why not just…?

Why not just query the existing exact index with many n-grams from each item? This is the most serious threat. It is exactly our first ablation: raw-index anchors with count-based selection. If it matches the full method, the canonical index is not worth its doubled build cost, which Infini-gram mini reports as 99 days single-node for 83TB. The paper would then recommend the cheaper procedure. We expect the gap to show up mainly on number-perturbed and reformatted items. That is where raw byte matching breaks every short span containing a digit or a markup token.

Why not just build a compressed dense vector index over the corpus? Dense retrieval is the only option here that handles translation, and we report that win openly. But a vector per passage at web scale, plus the redundancy that metadata filtering adds, is costly. Window-filtered nearest-neighbor search reports 3× to 14× storage inflation for restricting search to metadata ranges such as crawl dates. Quantization lowers recall. The matched-footprint comparison measures directly how much recall survives at the storage our method uses.

Why not just test the model instead of the corpus, with MIN-K% PROB? MIN-K% PROB flags text on which a model is unusually confident. It needs token log-probabilities. More importantly, the fragility study finds that one epoch of reinforcement-learning fine-tuning (GRPO) pushes reference-free detectors toward 50% AUROC. It also finds that chain-of-thought contamination leaves ten detectors at 50–58%. Corpus search does not depend on the final weights, which is why we pursue it. We will also run MIN-K% on a small model continued-pretrained on our planted slice, to show the contrast on identical plants.

Why not just use fresh benchmarks like LiveCodeBench? Time-stamped benchmarks such as LiveCodeBench avoid future leaks but say nothing about the static benchmarks still reported everywhere. They also depend on automatic verifiers. Auditing remains necessary for MMLU, GSM8K and HumanEval.

Proposed Experiments

The development slice is a few hundred gigabytes drawn from DCLM-baseline and FineWeb, with the plants inserted. At that size, uncompressed dense retrieval is feasible and every method can be compared. Benchmarks are GSM8K, MMLU and HumanEval. Metrics are:

  • recall at fixed judge precision, broken down by plant type;
  • candidates per item sent to the judge;
  • disk footprint and RAM per query;
  • query latency and build time.

The first decisive check is the per-type recall table against matched-footprint dense retrieval and the two ablations.

The second check runs at scale: the full canonical index over DCLM-baseline (17TB). The question is how far the MMLU dirty rate rises above the 27.7% exact-match figure, and how many GSM8K items have number-perturbed copies. We will hand-audit a stratified sample of new hits so the reported rate is not inflated by judge errors.

The third check asks whether new detections matter. A small model continued-pretrained on the planted slice should show higher per-item accuracy on items our method newly flags than on items it clears. If newly flagged items show no accuracy lift, the extra recall is detecting overlap that does not inflate scores, and we would say so.