When a language model's training data contains test questions from a benchmark, its score on that benchmark stops measuring ability. This problem is called benchmark contamination. Work on contamination asks two things: how many times a model must see a leaked item before the item inflates its score, and whether an outside auditor can tell from the model's token probabilities that the item was seen. Both answers have been measured with one tokenizer held fixed. A tokenizer is the component that splits text into vocabulary units called subwords. Vocabulary size has meanwhile become a free design choice. Recent models run from tens of thousands of subwords to half a million, and larger vocabularies have been shown to raise benchmark scores. That raises a question nobody in this literature has tested. A model with a 500k-entry vocabulary may encode a leaked question as twenty dense tokens instead of a hundred fragments. If so, it may memorize and exploit that question after far fewer exposures. The low-probability tokens that detectors rely on would then also come from a different part of the distribution.
This note proposes a controlled study of that question. We will pretrain matched families of models that differ only in vocabulary size and inject benchmark items at known repetition counts and known points in training. For each vocabulary size we will measure three things: the repetition count at which memorization turns into score inflation, how detection accuracy and false-positive rates for standard probability-based detectors shift, and whether summing token probabilities onto fixed character spans removes that shift. The practical stake is direct. Detection thresholds are calibrated on one model family and then applied to models with very different tokenizers. Vocabulary-scaling gains, such as the rise from 49.4 to 58.6 average Japanese accuracy between 5k and 500k vocabularies reported in Large Vocabulary Size Improves Large Language Models, are credited entirely to compression. Nobody has checked whether part of that gain is easier memorization.
Core Claim
We hypothesize that vocabulary size is a hidden variable in contamination, acting through two separate channels. The first is exploitation. At a fixed number of exposures and a fixed amount of training text, larger vocabularies lower the repetition count at which a leaked item raises benchmark accuracy. The second is detection. Token-level detectors are calibrated in units of tokens, and their false-positive rates do not carry over across vocabulary sizes. For Min-K% PROB, the token units are the "lowest-probability k% of tokens". For Min-K%++, they are the "z-score against the whole vocabulary". A threshold tuned at 32k will therefore over-flag or under-flag at 2k or 500k.
The new contribution has three parts. The first is the controlled factorial design that treats vocabulary size as an experimental factor in contamination. The second is a separate test of a third mechanism, tokenizer contamination: the tokenizer is itself trained on contaminated text, so benchmark phrases become dedicated vocabulary entries. The third is a tokenizer-invariant way to score detections, which we test as a remedy.
Either half of the claim can fail independently. One possible outcome is that exploitation curves line up once exposure is measured in bytes. Then vocabulary size only rescales sequence length, and the audit community can safely ignore it after byte normalization. That would be a useful negative result, and the design is built to detect it.
Relation to Prior Work
The motivating evidence comes from Data Contamination: From Memorization to Exploitation. That paper pretrained BERT on up to 600M Wikipedia tokens with injected downstream data. It found that single exposures produced about 15% memorization but under 1% exploitation, that 200 duplications pushed exploitation to 40%, and that early contamination was more harmful than late contamination. Its tokenizer was a fixed 30k WordPiece vocabulary in a masked model. We reuse its separation of memorization from exploitation and its duplication-and-timing design. We move the design to autoregressive models and make vocabulary size the main factor.
The vocabulary-scaling literature supplies the mechanism we expect to see. Finding the Optimal Vocabulary Size for Neural Machine Translation frames vocabulary choice as a tug-of-war between two pressures: shorter sequences ease autoregressive error accumulation, while larger vocabularies starve rare classes of training signal. It also proposes a rule that rare subwords need about 100 occurrences. That rule predicts that dedicated tokens for benchmark-specific strings would be poorly trained at low exposure counts. If so, large vocabularies would hinder rather than speed memorization. The two pressures make opposite predictions, which is why the experiment is worth running.
A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation shows that the best granularity depends strongly on data scale and architecture. It reports a 30.8 versus 27.5 BLEU gap between 500 and 32k merges in a low-resource setting. This constrains our design: conclusions drawn at one model scale may not carry over, so we sweep two model sizes. Large Vocabulary Size Improves Large Language Models supplies the vocabulary grid (5k to 500k, Unigram tokenization, SlimPajama). It also makes the claim we partly test, namely that larger vocabularies help because of compression alone.
On detection, Detecting Pretraining Data from Large Language Models introduced Min-K% PROB. The method averages the log-probabilities of the lowest-probability k% of tokens, on the idea that seen text lacks low-probability outliers. Min-K%++ z-normalizes each token's log-probability against statistics computed over the whole vocabulary at that position. Both methods are defined in token units, and both were tested only on models with standard vocabularies. Our proposal treats them as the objects under test, not as tools.
When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits supplies our statistical method. It shows that detectability scales with contamination fraction times behavioral separability times the square root of sample size. It also shows that naive Gaussian sample-size budgets fail and that a two-stage calibrated planner fixes them. We use its SQuAD-into-Pile injection protocol and its power analysis. We then ask whether its separability term depends on vocabulary size.
On The Fragility of Benchmark Contamination Detection in Reasoning Models is a constraint. It shows that a short round of reinforcement learning shrinks the probability gap that detectors exploit. Our study is therefore limited to pretraining contamination and makes no claims about post-trained reasoning models.
Generalization or Memorization detects contamination through peaked output distributions, using token-level edit distance between sampled outputs (CDD). That makes it another token-unit detector whose calibration we audit.
Research Program
We will train two model families. One has about 160M parameters and the other about 680M non-embedding parameters, to match the scale of the vocabulary-scaling study. Each family is trained at four vocabulary sizes, 8k, 32k, 128k and 500k, all built with the same Unigram algorithm on a SlimPajama sample. The corpus is first decontaminated against every benchmark we inject, using n-gram filtering.
Training is matched on bytes of text, not tokens. Every model sees the same documents in the same order, so the large-vocabulary models take fewer optimizer steps. We also run a secondary step-matched setting, so that "more steps" and "more information per step" can be separated. Embedding parameters grow with vocabulary size. We therefore report results at matched non-embedding size and add one tied-embedding control at 500k.
Into each run we inject disjoint shards of benchmark items at repetition counts of 1, 4, 16, 64 and 200. Each shard is inserted in either the first or the last fifth of training. We use three benchmarks. SQuAD questions with answers follow the injection protocol of the power-calibrated audits paper. GSM8K problems with solutions have a short-answer exploitation target. HumanEval problems allow us to compare against the rephrased-version gap reported in Generalization or Memorization. Each benchmark keeps a held-out clean split drawn from the same item distribution. Every configuration is run with three seeds.
For every model we measure three quantities, each against exposure count. Memorization is greedy exact-match completion of the answer given the prompt. Exploitation is few-shot accuracy on the injected shard minus accuracy on the clean split. Detectability is the detection score (AUROC) at a fixed 5% false-positive rate for each detector, computed per item between injected and clean items.
We will fit, for each vocabulary size, the exposure count at which exploitation crosses a fixed threshold. We report it as exposures per item, and separately as the number of times each item's tokens and bytes were seen.
We also run a tokenizer-contamination ablation at 128k. The model's injected data is held fixed, but the tokenizer is trained once on the clean corpus and once on the corpus with injected items present. This isolates whether gaining dedicated vocabulary entries for benchmark strings changes memorization by itself.
Finally, we build a tokenizer-invariant detector score. Each token's log-probability is spread over the characters it covers. We then apply the Min-K% and Min-K%++ selection over fixed 16-byte windows instead of over tokens. This lets one threshold be compared across tokenizers. The test is whether a threshold calibrated at 32k keeps its nominal false-positive rate at 8k and 500k.
Baselines
The naive reference point is perplexity thresholding, where items with lower average loss are flagged as seen, computed in each model's own tokens. It is the detector most auditors reach for first. Detecting Pretraining Data from Large Language Models reports that Min-K% PROB improves AUC by 7.4% over the previous best baseline on WikiMIA. So loss thresholding should be weak at standard vocabularies, and the interesting question is how its weakness changes across sizes. We run it on exactly the same items and models as every other detector, with no tuning.
The strongest existing reference-free method is Min-K%++. It reports beating the runner-up by 6.2% to 10.5% AUROC averaged over five models on WikiMIA. On MIMIR it reports parity with reference-based methods; no absolute MIMIR AUROC is reported in the file. We run Min-K%++ and Min-K% with k tuned on a validation split at 32k only. We then evaluate the 32k-tuned k at every vocabulary size, and separately with k retuned per vocabulary size under the same validation budget. Beating it means one of two things. Either our byte-window score keeps the false-positive rate within one percentage point of nominal across all four vocabularies where Min-K%++ does not, or it matches Min-K%++'s retuned AUROC without needing per-tokenizer retuning.
The reference-based detector compares the target model's loss to a small reference model trained on clean data with the same tokenizer. This is the strongest competitor on the calibration question, because a matched reference cancels much of the tokenizer's effect by construction. We train one reference per vocabulary size on the same clean bytes. It sets the upper bound that a reference-free, tokenizer-invariant score should approach.
For the exploitation question, the baseline is the curve from Data Contamination: From Memorization to Exploitation: under 1% exploitation at low repetition and 40% at 200 duplications, with a 30k WordPiece vocabulary. Our 32k configuration should roughly reproduce the shape of that curve in autoregressive form. If it does not, the comparison across vocabularies is suspect.
For audit power, the baseline is the two-stage planner of When Is Benchmark Contamination Detectable?, which reports 0.785 to 0.92 power against a 0.80 target. We ask whether a planner calibrated at 32k still delivers that power at other vocabulary sizes.
The key ablations are byte-matched versus step-matched training and clean versus contaminated tokenizer training.
Why not just…?
Why not just normalize everything by bytes and declare the problem solved? This is the most serious alternative, and it may be correct for exploitation. If the large vocabulary only shortens sequences, then exploitation curves plotted against bytes seen should overlap across vocabulary sizes. The byte-matched design tests exactly that. If the curves overlap within seed variance, we will report that vocabulary size does not matter for exploitation once exposure is counted in bytes. For detection, simple byte-length normalization of average loss does not address the problem, because Min-K%-style methods choose a subset of units, and which units fall in the bottom k% depends on segmentation. Our byte-window score is the principled version of this alternative. The ablation comparing it with plain per-byte loss shows whether the subset selection matters.
Why not just use a reference model with the same tokenizer? Where an auditor can build one, it may remove most vocabulary effects, and our reference baseline measures how much. Building one is often not possible. External auditors rarely have a clean model with the target's tokenizer, and contamination reports such as Contamination Report for Multilingual Benchmarks audit many model families with different tokenizers at once. If the reference method fully cancels vocabulary effects, the practical lesson becomes "always match tokenizers in the reference". That would still be a result the literature currently lacks.
Why not just compare existing open models that already use different vocabularies? Existing models with different vocabularies also differ in data, scale and training length. Any difference in contamination behavior would be confounded with all of those. The exchangeability test of Contamination Report for Multilingual Benchmarks shows contamination differs across Llama 3.1, Gemma 2 and Aya-23, but it cannot attribute those differences to tokenization. Only controlled injection with matched corpora can do that. We will still run our detectors on public models as an outside check, without causal claims.
Why not just evaluate on post-cutoff or dynamic benchmarks? Fresh benchmarks avoid contamination for evaluation. They do not tell model developers whether their vocabulary choice makes leaked data more harmful. They also do not tell auditors whether thresholds transfer across tokenizers. The survey in Benchmark Data Contamination of Large Language Models: A Survey notes that such benchmarks decay once published. Our study addresses the measurement problem that remains.
Decisive Checks
Four results would decide the study.
The first check concerns exploitation. Under byte-matched training, does the exposure count needed for exploitation on SQuAD and GSM8K differ between 8k and 500k by more than the spread across three seeds? A consistent shift in one direction would support the exploitation claim. Large vocabularies needing fewer exposures would favor the compression account. Needing more exposures would favor the rare-token account suggested by the 100-occurrence rule.
The second check concerns calibration. When the Min-K%++ threshold tuned at 32k is applied at 8k and 500k, does the realized false-positive rate depart from 5% by more than two points, and does the byte-window score close that gap? If token-unit detectors already transfer, the detection half of the claim fails.
The third check concerns tokenizer contamination. At 128k, with model-level exposure held fixed, does a tokenizer trained on contaminated text raise memorization at low repetition counts? A positive result would mean that the corpus used to build the tokenizer must also be decontaminated, which current practice does not do.
The fourth check concerns clean gains. On the decontaminated runs, does the clean-split accuracy gain from larger vocabularies persist? That tells us how much of the reported vocabulary-scaling benefit survives once contamination is ruled out by construction.
References
- Large Vocabulary Size Improves Large Language Models
- Data Contamination: From Memorization to Exploitation
- Finding the Optimal Vocabulary Size for Neural Machine Translation
- A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation
- Detecting Pretraining Data from Large Language Models
- Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
- Contamination Report for Multilingual Benchmarks
- Benchmark Data Contamination of Large Language Models: A Survey