Decontamination Is Only as Good as Its Embedding

Before a benchmark score can be trusted, someone usually has to answer a retrieval question: does anything in the training data say the same thing as this test item? Model developers deduplicate corpora against benchmarks, and benchmark curators prune redundant items. When n-gram overlap is too brittle, both groups reach for a sentence encoder, a model that turns each text into a vector so that nearby vectors mean similar texts, and they flag nearest neighbors. Representation-learning research has shown that algorithms run on top of an embedding inherit its blind spots: no clustering or selection rule can recover a distinction the encoder threw away. This proposal applies that lesson to contamination. Our claim is that embedding-based decontamination is capped by whether the encoder represents the answer-determining content of a test item (its numbers, constraints, and logical structure) rather than its surface story. We expect off-the-shelf encoders to fail in both directions. They will miss paraphrased leaks that keep the math and change the narrative, and they will over-flag genuinely different problems that happen to share a story.

The opportunity matters because the hardest contamination is no longer verbatim. Rewrites in the style of Inference-Time Decontamination, which turns a sprinting word problem into a swimming one while keeping its logic and numbers, are exactly the leaks a story-dominated embedding would place far apart. Once such data reaches training, the model-side detectors that might catch it after the fact fail on reasoning models, as On The Fragility of Benchmark Contamination Detection in Reasoning Models shows. Data-side filtering is therefore the last line of defense, and nobody has measured how much the choice of encoder limits it.

Core Claim

We make two testable claims. The first is that the residual score inflation left after embedding-based decontamination is predicted by a specific, cheaply measured property of the encoder. Call it answer-content separability: on a small set of labeled triplets, how often a paraphrase that preserves the answer sits closer to the original than a near-miss that keeps the story but changes the numbers or constraints. We further predict that generic, task-agnostic quality measures predict this poorly. Effective rank (how many directions of the embedding space actually carry variance) and the spectral entropy used in Bures-Isotropy Alignment are examples of such measures.

The second claim is that a lightweight contrastive adaptation of the encoder closes most of the gap. The adaptation uses answer-preserving rewrites as positives and story-preserving near-misses as hard negatives. We expect it to beat simply swapping in a larger general-purpose encoder, and to beat a generic isotropy regularizer.

The second half of the first claim speaks directly to an open problem in the representation-quality literature: whether representation quality can be judged apart from the task. In this setting we predict that it cannot be judged usefully that way.

Relation to Prior Work

The representation-bottleneck line motivates the mechanism. Several papers report that downstream selection algorithms fail when the extractor misses semantic dimensions: optimal-transport coreset selection (Weiwei Xiao), similarity-matched synthetic data acquisition (Niemeijer et al.), and experimental-design data markets (Data Acquisition via Experimental Design for Data Markets). Bures-Isotropy Alignment shows that compression-style objectives concentrate feature energy in a few directions, and it restores rank by maximizing the nuclear norm (the sum of singular values) of the class tokens. We borrow its rank-restoring regularizer, but only as an ablation. Our hypothesis is that a generic isotropy fix spreads variance without aligning it to answer content.

On the contamination side, the closest work is SMART filtering. SMART prunes semantically redundant benchmark items using SentenceBert distances and detects contamination with a no-context multiple-choice test. It never asks whether SentenceBert is the right geometry, and its contamination test cannot extend to open-ended math or code. We keep its pipeline fixed and vary only the encoder, and we target generative benchmarks where its contamination test does not apply.

Rethinking Benchmark and Contamination for Language Models with Rephrased Samples motivates the threat model: rephrasings evade surface matching. We go further and ask which embeddings also miss them, and why.

The model-side detectors constrain our scope rather than compete with it. LaRA, CANARY, and RADAR read a trained model's hidden states. They need white-box access and, in two of the three cases, a clean reference model or a clean calibration set. Our audit runs on the corpus before training, needs no reference model, and serves the developer who controls the data. The Fragility result motivates why this upstream position matters.

Research Program

Triplet set. We first build a ground-truth triplet set on GSM8K and on the competition-math sets used by the Fragility and LaRA papers (AIME, AMC, OlympiadBench, Minerva Math). Each test item gets two kinds of companion text.

Answer-preserving rewrites keep the item's numbers and logic and change its story. Some are produced ITD-style by an LLM, and some are produced by programmatic templating. Story-preserving near-misses keep the narrative but alter a number or a constraint so that the answer changes.

Every label is checked by solving the variant. A rewrite must reproduce the original answer, and a near-miss must not. This step matters because the labels must not inherit the rewriter's own blind spots.

Separability score. On this set we compute answer-content separability for each candidate encoder. Separability is the fraction of triplets in which the encoder places the original item closer to its answer-preserving rewrite than to its story-preserving near-miss. Alongside it we compute effective rank and spectral entropy on the same texts.

Controlled contamination. We continue-train a small open model on a background corpus sampled from FineWeb. Into it we inject benchmark items at known rates, using three variant types: verbatim copies, answer-preserving rewrites, and near-misses. Near-misses are deliberately harmless, because removing them costs useful data without reducing leakage.

Each encoder then runs the same nearest-neighbor decontamination at a matched removal budget. We retrain on each filtered corpus and measure residual inflation. Residual inflation is the benchmark score minus that of a model trained on the known-clean corpus with the same seed and token count. A score that is still inflated means leaked rewrites survived the filter.

Encoder adaptation. Finally, we fine-tune a sentence encoder contrastively on held-out triplets. The rewrites serve as positives and the near-misses as hard negatives, and no benchmark used for evaluation appears in this training set.

Baselines

The naive reference point is no decontamination. It fixes the ceiling of inflation in our setup. For calibration, the Fragility paper reports that SFT-stage contamination inflated pass@1 by an average of 8.8% on its competition-math suite. We should see inflation of that order before filtering, or our injection rate is too weak to be informative.

The surface baseline is n-gram overlap filtering. It is the standard corpus-side practice and the cheapest competitor. We run it at the same removal budget as every other filter, so that differences reflect which items get flagged rather than how many.

The strongest existing embedding method is the SMART pipeline with its SentenceBert encoder. On ARC, SMART reports raising Pearson correlation with ChatBot Arena Elo from 0.783 to 0.845 and removing 68.9% of items, while keeping Kendall's Tau above 0.95. We rerun SMART unchanged on ARC, MMLU, and CommonsenseQA and swap only the encoder. To count as beating it, the adapted encoder must either preserve or raise the 0.845 correlation while pruning fewer items that solving shows to be distinct, or yield lower residual inflation on the math sets at equal budget.

We add a larger general-purpose embedding model to separate the effect of the training objective from the effect of scale. We also add retrieve-then-judge, in which an LLM checks each embedding-retrieved candidate pair. Every encoder gets the same candidate pool, the same retrieval depth, and the same tuning budget for its distance threshold, chosen on a held-out split.

Two ablations isolate the key ingredient. The first adapts the encoder with positives only, which tests whether the near-miss hard negatives matter. The second applies the BIA nuclear-norm regularizer with no task pairs, which tests whether generic rank restoration is enough.

As a model-side reference rather than a competitor, we report Min-K% and LiRA on the trained models. The Fragility paper puts these detectors at roughly 50–58% AUROC for CoT-contaminated reasoning models.

Why not just…?

Why not just use n-gram matching, which is cheap and standard? An answer-preserving rewrite by construction shares the numbers and few of the content words. The Rephrased Samples work argues that such leaks evade surface matching, and EvoEval-style rewriting cut 51 models' HumanEval scores by an average of 39.4% according to the BDC survey. Our residual-inflation measurement tests this directly: if n-gram filtering removes the inflation from rewrites, the embedding question is moot.

Why not just use a bigger embedding model? This is a real threat. If scale alone produces high answer-content separability, the adaptation step adds little. The large-encoder baseline settles it. Our prediction is that general encoders are trained to cluster texts by topic and paraphrase, which rewards story overlap. If the large encoder matches the adapted one, we will report that the bottleneck is a matter of scale rather than of objective.

Why not just retrieve candidates with any embedding and let an LLM judge decide? A judge can only fix precision. It never sees leaks that the embedding failed to retrieve. We therefore report candidate recall at fixed retrieval depth for each encoder. If retrieve-then-judge recovers most leaks even with SentenceBert at modest depth, the bottleneck is weaker than we claim, and the cheaper fix of retrieving deeper wins.

Why not just evaluate on fresh competitions? Fresh competitions avoid contamination, but MathArena gathered only 162 problems in a year, at about $53 per model on IMO 2025 alone. That supply cannot cover training-time decisions or the backlog of existing benchmarks, so fresh competitions complement corpus filtering rather than replace it.

Decisive Checks

The proposal stands or falls on three results. First, across all encoders, answer-content separability should rank-correlate with residual inflation after filtering, and it should do so more strongly than effective rank or spectral entropy. If the generic measures predict just as well, the task-specific claim fails.

Second, the adapted encoder should reduce residual inflation from rewrite-type leaks below the SentenceBert and n-gram filters at equal budget, and it should prune fewer near-misses.

Third, the hard-negative ablation should account for most of that gain, and the isotropy-only ablation should not. If the regularizer alone suffices, the bottleneck is generic collapse rather than misaligned content, and we would revise the claim accordingly.