Recent work shows that the kind of web document a language model is trained on (its register, or genre: opinion blogs, news, how-to guides, encyclopedic description, FAQs) strongly predicts how well the model scores on standard benchmarks. Mixtures of a few good registers even beat carefully filtered web corpora. The benchmarks that produced these rankings are themselves on the web, though, and they are unlikely to be spread evenly across genres. A science quiz question is far more likely to sit on a homework-help page or an explanatory article than in song lyrics or a news story. If so, some register "utility" may be leaked test material rather than better language for learning.
We propose a controlled study that retrains register-specific models after removing benchmark-related material from each register, and then tests them on rewritten versions of the benchmarks. A gain that survives both steps reflects what the register teaches. A gain that disappears was leakage. The point is not to tally overlaps per register, which is cheap and does not settle the question. The point is to measure how much of each register's measured benefit is causally due to contamination. That answer decides whether register-based data mixing is a sound curation principle or partly an artifact of how we evaluate.
Core Claim
We hypothesize that register rankings on multiple-choice benchmarks mix two effects: a real effect of the register's language, and a contamination effect concentrated in instructional, descriptive and question-answer registers. Our prediction is directional and can fail. The advantage of "How-to/Instructions" and "Informational Description" should shrink measurably under semantic decontamination and on rewritten test items. The advantage of "Opinion" should largely survive, because opinion pages rarely restate quiz items.
If every register's ranking survives decontamination and perturbation, register utility is real and contamination is a minor confound. That is also a useful, publishable outcome.
Where This Sits
The motivating evidence comes from Register Always Matters. That paper trained small models on individual registers of deduplicated English HPLT v2 and found:
- Opinion ranked 4th among single registers.
- News underperformed its parent register Narrative.
- Mixtures such as HI-IN-HI-dtp-OP beat both full HPLT v2 and FineWeb.
The authors flag benchmark contamination as an untested limitation. We test exactly that.
The FineWeb Datasets raises the same unresolved question from the other side. Its educational-value classifier, effectively a genre filter for textbook-like pages, lifts ARC from 46% to 57% and MMLU from 33% to 37%. It never separates how much of that lift comes from benchmark-adjacent pages.
Building Question-Answer Data Using Web Register Identification supplies a method we reuse: a cheap document-level register classifier acts as a gate in front of expensive token-level processing. It reports precision 0.82 and recall 0.47 on the QA class. It uses the gate to harvest QA data and never asks whether that harvest is enriched with benchmark items. We use the same gating idea in reverse, to direct an expensive contamination check at the registers most at risk.
Three contamination papers constrain the design.
- Causally Quantifying the Effect of Test Set Contamination on Generative Benchmarks injects MATH replicas uniformly into FineWeb-Edu. It shows that inflation grows with replica count and model size, and that it is masked by high sampling temperature. We borrow its injection design but vary the register that wraps the injected item. We also treat its finding as a warning that small models may show weak contamination effects.
- Synthetic Data and the Rise of Spiky Intelligence shows that benchmark-adjacent text, without verbatim copies, inflates scores. It measures this with an Inflation Score on surface-rewritten items (Factory-GSM). It also reports that embedding similarity is a weak predictor of which items gain (AUC 0.54 versus 0.68 for gradient similarity). That result directly constrains our decontamination step: plain embedding retrieval will not be enough on its own.
- Proving Test Set Contamination in Black-Box Language Models gives a statistical probe. A contaminated model prefers the canonical order of benchmark items over shuffled orders. The test detected 2 to 4 exposures in a 1.4B model and flagged Mistral-7B on ARC at $p=0.001$.
Research Program
Corpus and stratification. We start from the deduplicated English HPLT v2 portion (HPLT v2), labelled with the same multi-label register classifier used by Myntti et al. so that our numbers are directly comparable. For a sample of documents in each register, we also record the FineWeb-Edu educational score. This lets us check whether register adds anything beyond educational-ness.
Two-tier decontamination. We build a benchmark-adjacency detector for the LightEval multiple-choice suite (ARC Easy and Challenge, OpenBookQA, CommonsenseQA, PIQA, SIQA, HellaSwag, WinoGrande; MMLU is reported but not relied on, since small models sit near chance on it).
- Verbatim tier. Standard 13-gram overlap removal.
- Semantic tier. Embedding retrieval proposes candidate documents near any test item. A cross-encoder, or an LLM judge, then decides whether each candidate restates, paraphrases or answers a specific test item.
Because embedding similarity alone is weak, the judge is calibrated on held-out injected paraphrases and translations. We report its recall at fixed precision. The judge is too expensive to run everywhere. Following the gating idea from Eskelinen et al., we run it exhaustively on QA, FAQ, instructional and descriptive registers, and on a uniform random sample of all other registers. The sample gives an unbiased contamination-rate estimate everywhere, so the gate saves compute without hiding contamination where we did not expect it.
Retraining arms. For each major register, we train models matched to Myntti et al. in size, token budget and evaluation harness, under four conditions:
- Original: the register as Myntti et al. used it.
- N-gram: after verbatim decontamination.
- Semantic: after full semantic decontamination.
- Random removal: the same number of documents removed as in the semantic arm, chosen at random from the register.
The random-removal control separates the effect of losing benchmark-related content from simply losing data. We repeat the four arms for the two headline mixtures, HI-IN-HI-dtp-OP and HI-IN-HI-dtp-OP-NA, and for the full-corpus baseline. With multiple seeds for the top registers, this comes to a few dozen small-model runs.
Contamination-robust evaluation. Every model is evaluated three ways:
- Original benchmarks.
- Rewritten items. For ARC, OpenBookQA and CommonsenseQA, which have no clean parallel sets of their own, we build Factory-GSM-style rewrites that change surface entities and wording while keeping the tested fact and the answer, and we reorder the options. Rewrites are spot-checked by humans for answer preservation.
- Group-scored WinoGrande. Following Back to Square One, credit requires resolving both twin sentences correctly. That paper showed group scoring removes much artifact-driven accuracy: the top WSC model fell from 89.71% to 79.41%, and zero-shot masked language models dropped to random on WinoGrande.
The central quantity is register-level inflation, adapted from the Inflation Score. For a register $r$, let $\Delta^{\text{orig}}_r$ be its accuracy gain over a random-web reference on the original benchmarks, and $\Delta^{\text{rew}}_r$ the same gain on rewritten items. Then
$$I_r = \Delta^{\text{orig}}_r - \Delta^{\text{rew}}_r .$$
A register whose benefit is real should have $I_r$ near zero, and its decontaminated gain should stay close to its original gain.
Register as a context for memorization. As a smaller, supporting experiment, we inject a fixed set of held-out benchmark items at controlled replica counts into otherwise clean training data, wrapped in documents from different registers (forum thread, tutorial, opinion post). We then measure two things: accuracy gain on the injected items and the order-preference statistic of Oren et al. This calibrates whether our probes can see contamination at our model scale at all. It also tests whether register wrapping changes how quickly a model absorbs a test item.
Baselines
Naive reference. The do-nothing reference is the original register ranking and mixture result of Myntti et al., reproduced in our harness. If decontaminated rankings match this reference within seed variance, the proposal's hypothesis fails. That counts as a result, not a nuisance.
N-gram decontamination. The strongest existing decontamination method in common use is 13-gram removal, the default in large-corpus pipelines and the main matching-based method in the contamination survey. It runs on the identical corpus slice with the identical training budget, so the only difference from the semantic arm is which documents are removed. On the detection side, the strongest reported comparison is from Generalization or Memorization: distribution-based detection improved accuracy and AUC by 21.8% to 30.2% relative to n-gram baselines on implicit leaks. No reported result found on Lacuna for the effect of n-gram decontamination on register-level benchmark rankings.
Semantic decontamination "beats" this baseline if both of the following hold:
- It removes documents whose deletion causes a statistically significant drop in a register's original-benchmark gain, beyond what random removal of the same volume causes.
- That drop is concentrated on the items the removed documents matched.
Educational-score stratification. To show that register explains something educational-ness does not, we compare against selecting by FineWeb-Edu classifier score at matched token counts. FineWeb-Edu itself reports ARC rising from 46% to 57% (FineWeb). Register "wins" here only if, within educational-score bins, register still predicts rewritten-item accuracy.
Key ablations. Random removal at matched volume isolates the content-specific effect of decontamination. A register-blind version of the semantic filter, run at the same total compute budget but sampling uniformly rather than gating by register, tests whether register gating actually improves recall per unit of compute.
Why not just…?
Why not just count benchmark overlaps per register and skip retraining? Prevalence is not effect. The work on causal contamination (Schaeffer et al.) shows that inflation depends nonlinearly on replica count, model scale and item difficulty. A register could hold many near-duplicates yet contribute little gain, or hold few that matter a lot. We do produce the overlap census, since it costs little, but only retraining with the random-removal control answers whether register utility is contaminated.
Why not just apply standard 13-gram scrubbing and rerun Myntti et al.? This is a real threat to the proposal, so it is one of our arms. If the n-gram arm already removes register differences in inflation, the semantic tier is unnecessary and we will say so. Our prior that it will not suffice comes from evidence that rephrased and benchmark-adjacent text slips past n-gram filters (Dong et al.; Spiky Intelligence). The comparison between the n-gram and semantic arms settles it.
Why not just evaluate on post-cutoff or dynamic benchmarks? For small models trained on one corpus, dynamic suites mostly target generative coding and math, where these models score near zero, so they give no signal about registers. They also cannot say which part of the training data caused an inflated score. Rewritten items and group scoring give a contamination-robust view of the same skills the original rankings measured, which is the comparison we need.
Why not just use the educational-quality classifier instead of registers? Register may be a proxy for educational-ness. The educational-score stratification baseline tests that directly. If register adds nothing within educational bins, the honest conclusion is that register-based mixing reduces to quality filtering for these benchmarks.
Decisive Checks
The study succeeds as science whichever way the result falls, provided three checks pass:
- The calibrated judge recovers injected paraphrases at a reported recall.
- The order-preference probe detects our own injected contamination at our model scale. Without this, null results are uninterpretable, and we would scale the models up rather than claim a null.
- Rewritten items are human-verified to preserve answers.
The decisive comparison has two parts:
- Do How-to and Informational Description show positive $I_r$ and a decontamination drop exceeding random removal, while Opinion does not?
- Does HI-IN-HI-dtp-OP still beat the full-corpus baseline after semantic decontamination and on rewritten items?
A yes to the second confirms that register mixing is a sound curation tool. A no means that part of the published register advantage was benchmark leakage.
References
- Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Building Question-Answer Data Using Web Register Identification
- Causally Quantifying the Effect of Test Set Contamination on Generative Benchmarks
- Synthetic Data and the Rise of Spiky Intelligence
- Proving Test Set Contamination in Black-Box Language Models
- An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
- Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema
- Benchmark Data Contamination of Large Language Models: A Survey
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models