When a language model has seen a benchmark's test questions during training, its score mixes two things: real skill and recall of answers it has already seen. For translated leaks, where a model trained on French copies of English test questions and then takes the English test, the usual detectors fail. String-overlap filters find nothing, likelihood checks miss most cases, and the one test that works is limited to multiple choice. The best black-box audits say only whether a benchmark leaked, not how many points the leak is worth. This note proposes a way to measure that number by compressing the model. We quantize its weights (store them with fewer bits) using a bit allocation calibrated only on ordinary text, then watch which benchmark answers disappear first.
The intuition comes from rate-distortion theory, the study of how much information survives a limited bit budget. Memorized, item-specific answers are expensive to store and are not needed to model ordinary text. General skills share capacity with everything else. A compressor that spends its bits protecting general text should therefore erase leaked answers before it erases general skills. The benchmark points lost inside that window, beyond what a matched clean control loses, estimate contamination in points. The estimate works for multiple choice, extractive QA, and chain-of-thought alike, and does not care which language the leak came through. By varying which languages the calibration text covers, the same probe can also point to the language channel the leak used.
Core Claim
Our claim is that a bit-allocating quantizer, calibrated on generic multilingual text, has a range of bit budgets we call a decontamination window. Inside it, general ability on held-out, uncontaminated data stays within about two points of full precision, while accuracy on leaked items falls toward the level of a clean model. The size of that differential collapse gives a continuous, per-benchmark estimate of contamination-inflated points. Let $a_B(r)$ be benchmark accuracy at average bit-width $r$, $a_C(r)$ accuracy on a difficulty-matched control set the model has not seen, and $r^*$ a budget inside the window. The estimate is a difference-in-differences:
$$\hat{\Delta} = \big[a_B(16) - a_B(r^*)\big] - \kappa\,\big[a_C(16) - a_C(r^*)\big],$$
Here $\kappa$ rescales the control's drop to the benchmark's difficulty. We fit $\kappa$ on an uncontaminated reference model, where $\hat{\Delta}$ should be zero.
The idea is not "compression removes contamination" in general. The claim is narrower and testable. First, the leak must live in weights that generic-text calibration does not protect. Second, this holds whether the leaked copy was in the test language or a translation of it.
Where This Sits in Prior Work
Three papers motivate the gap. Data Contamination Can Cross Language Barriers contaminates LLaMA3-8B and Qwen1.5-7B with translated MMLU, ARC-Challenge and MathQA. It shows 5–10% MMLU inflation that n-gram and likelihood checks miss, and MathQA rising from 42.01% to over 95%. Its detector, Choice Confusion, swaps in distractors taken from other items and works only for multiple choice. Contamination Report for Multilingual Benchmarks tests whether a model prefers the benchmark file's original item order over random permutations. It flags Llama-3.1-8B and Gemma-2-9B on nearly all multilingual benchmarks ($p < 0.0013$), but it gives only a yes/no verdict and cannot say whether the leak was English or translated. Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models shows that "zero-shot" cross-lingual transfer, meaning a model trained on one language performing in others, partly tracks accidental foreign-language text in pretraining. It correlates that text volume with downstream performance ($\rho$ up to 0.67) but does not study leaked test items. Our proposal adds what all three lack: a graded, format-agnostic measurement plus a provenance signal.
Two papers supply the method. Radio: Rate–Distortion Optimization for Large Language Model Compression chooses per-group bit depths by stochastic dual ascent on output error over a 128-example calibration set. The resulting bit depth is logarithmic in each group's sensitivity. That makes explicit what the calibration text protects, and the calibration set becomes our experimental knob. Radio used only C4 and English benchmarks, and it never asked what compression does to memorized test data. Principled SVD-based Delta Compression via Quantization Error Minimization (PrinMix) compresses the fine-tuning delta by solving an integer program over bit-widths for singular vectors. We use it for a second setting, where contamination enters during fine-tuning so the leak sits in the delta.
Three papers constrain the design and sharpen the predictions. Hallucination is a Consequence of Space-Optimality proves that storing unstructured facts has a rate cost set by how well the model separates facts from non-facts, and shows transformers operate within 12–22% of that bound. That supports the prediction that leaked answers are costly and are shed first. The theory covers only unstructured keys, so we treat it as motivation, not a derivation. Data Contamination: From Memorization to Exploitation shows that exploitation grows with duplication (up to 40% at 200 copies). That warns that heavily duplicated leaks may be stored robustly enough to survive compression, so the window must be measured as a function of leak dose. A Narrowing Geometry in Contaminated Reasoning finds that contaminated reasoning concentrates in a few activation directions, and that projecting them out restores reasoning on about half of leaked GSM8K/MSCalc items. That is independent evidence that leaks occupy a small, separable part of the model. It is also our main mechanistic competitor, tested only on English arithmetic.
Research Program
We first build models with known contamination, so the true inflation is known item by item. Following Yao et al.'s protocol, we continue pretraining LLaMA3-8B and Qwen-family 7B models on translated test sets. The benchmarks are MMLU, MathQA and ARC-Challenge for multiple choice, MLQA (MLQA) for extractive QA, and GSM8K for chain-of-thought. For each benchmark we leak a random half of the items, which lets every item-level detector be scored. We cross three factors: leak language (French, German, Spanish vs. Chinese, Japanese, Korean, to test the asymmetry Yao et al. report); duplication count (1, 10, 100); and whether the leak is English or translated. Uncontaminated checkpoints trained on the same volume of generic text serve as references and are used to fit $\kappa$.
Second, for each model we sweep Radio at 2, 2.5, 3, 4 and 16 bits with calibration on generic multilingual text. At each budget we record benchmark accuracy on leaked and non-leaked halves, accuracy on difficulty-matched controls, and perplexity on held-out multilingual text. The control set is non-leaked items plus freshly rewritten variants with the same answer type. We report $\hat{\Delta}$, the width of the window, and item-level detection quality.
Third, we run a calibration-language contrast for provenance. We repeat the sweep with calibration on English only, and then with English plus each leak-candidate language. If the leak is stored through a French channel, adding French to the calibration should protect it. Inflation that survives only under French-inclusive calibration then points to French. We score this against the known leak language.
Finally, we apply the calibrated probe to released models. We use Llama-3.1-8B and Gemma-2-9B on PAWS-X and XCOPA, which Ahuja et al. flag as contaminated. Aya-23-8B, which passes the order test on those same benchmarks, is the in-the-wild negative reference. This yields the first point estimates of how much those flagged scores are inflated.
Baselines
The naive reference is the raw gap between accuracy on original and rewritten items at full precision, with no compression. It is what a practitioner would compute first. It is confounded because rewriting changes difficulty, and it gives no item-level or provenance signal. We run it on the same rewritten controls, and the probe must beat it on inflation-estimate error in the controlled models.
The strongest format-specific competitor is Choice Confusion from Data Contamination Can Cross Language Barriers. It reports that on MMLU a clean LLaMA3-8B gains +26.25 points under the distractor swap, while a contaminated one drops about 17 points. We re-run it on our exact contaminated checkpoints for MMLU, MathQA and ARC-Challenge with identical prompts. Beating it means matching its benchmark-level detection on multiple choice while also giving calibrated point estimates. It cannot produce estimates at all on MLQA and GSM8K, so extending coverage to those formats counts as the second half of beating it.
The strongest black-box auditing competitor is the canonical-order exchangeability test from Contamination Report for Multilingual Benchmarks, which reports $p < 0.0013$ for flagged models. It needs no weights, which is a real advantage. We run it with the same permutation budget on every checkpoint. Beating it means detecting low-dose translated leaks it misses, plus the graded and provenance outputs it cannot give. If it detects everything we detect, our contribution narrows to quantification.
The strongest mechanistic competitor is Top-Subspace Suppression from A Narrowing Geometry in Contaminated Reasoning. On GSM8K/MSCalc it reports an 11.03% absolute consistency gain over prior baselines, with under 6% clean-data loss. We implement its eigenvector projection on the same models, with a matched budget for choosing the number of projected directions. We then turn its per-item recovery into an inflation estimate. The probe beats it if it gives lower estimate error off arithmetic tasks and across languages, where the geometry has not been tested.
Two ablations isolate the key ingredients. The allocator ablation replaces Radio with round-to-nearest and with GPTQ at matched average bits and identical calibration data. For reference, Radio reports GSM8K 49.81% vs. GPTQ 46.47% vs. round-to-nearest 6.14% at 3 bits on Llama-2 70B. The calibration ablation replaces generic calibration text with text that includes benchmark-domain material. Our hypothesis predicts that the window shrinks because the leak becomes protected.
Why not just…?
Why not just compare scores on the original and freshly rewritten items? The file contains no reported result on Lacuna for such a rewritten-item gap on translated leaks, and rewriting always shifts difficulty. A drop could mean contamination or a harder paraphrase. The difference-in-differences design exists to divide out that shift. The controlled experiment settles the question directly: if the naive gap estimates the known inflation as well as $\hat{\Delta}$ does, compression adds nothing and we will say so.
Why not just use Choice Confusion or the order test? Both are good detectors for what they cover. Choice Confusion needs multiple-choice distractors. The order test needs the benchmark to have leaked as an ordered file, and it answers only yes or no. The open problem Ahuja et al.'s own results leave is exactly how many points a leak is worth and through which language it came. Our experiments measure both against ground truth, where neither baseline produces an answer.
Why not just project out the memorization subspace? TSS is the closest threat, because it rests on the same premise that leaks occupy a small separable region. It is validated only on English arithmetic, in one module type. It also needs per-model eigen-analysis on leaked examples, which presupposes knowing which items leaked. If TSS-derived estimates match ours on MLQA and translated MMLU, the two methods are measuring the same thing. The result would then favor whichever is cheaper, and we report that comparison directly.
Why not just sweep plain GPTQ or uniform rounding? This is a genuine threat to the rate-distortion part of the claim. The probe's logic depends on calibration choosing what to protect, not on Radio specifically. If GPTQ with generic calibration yields the same window, the finding becomes "calibrated compression measures contamination" and we drop the claim that sensitivity-weighted allocation matters. Uniform rounding, which ignores calibration, is predicted to erase general skill and leaks together, leaving no window. If it does leave a window, that would falsify the mechanism we propose.
Decisive Checks
The first check is whether a window exists. On controlled models, some bit budget must keep held-out perplexity and control accuracy within two points of full precision while cutting leaked-half accuracy by at least half the inflation. If no such budget exists at any leak dose, the idea fails.
The second check is calibration. Across the grid of benchmarks, languages and doses, $\hat{\Delta}$ should track true inflation with estimate error below the naive rewritten-item gap. On uncontaminated reference models it should be near zero, which is the false-positive test. Item-level scoring of leaked vs. non-leaked halves should beat Choice Confusion on multiple choice and should work at all on MLQA and GSM8K.
The third check is dose and language. We expect the window to narrow as duplication rises, as Magar and Schwartz's exploitation results suggest. We also expect the calibration-language contrast to recover the true leak language above chance, and more reliably for European languages, following Yao et al.'s asymmetry.
The last check is in-the-wild coherence. The estimated inflation for Llama-3.1-8B and Gemma-2-9B on PAWS-X and XCOPA should be clearly positive, and Aya-23-8B's should be near zero, agreeing with the order test's verdicts. We then attach, for the first time, a number to how much of those scores the leak explains.
References
- Data Contamination Can Cross Language Barriers
- Contamination Report for Multilingual Benchmarks
- Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models
- Radio: Rate–Distortion Optimization for Large Language Model Compression
- Principled SVD-based Delta Compression via Quantization Error Minimization
- Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testing
- Data Contamination: From Memorization to Exploitation
- A Narrowing Geometry in Contaminated Reasoning
- MLQA: Evaluating Cross-lingual Extractive Question Answering