When a released model fine-tuned from an open base scores well on a public test, a reader cannot tell whether the gain is a real skill or a memorized copy of the test. This proposal uses a known quirk of low-precision deployment to separate the two. Quantization stores each weight as one of a few allowed values, which we call bins. A recent result showed that the small weight changes made while "unlearning" data mostly fall inside a single bin. When the model is quantized, those changes are rounded back to the original pre-trained values, and the supposedly forgotten knowledge returns (Catastrophic Failure of LLM Unlearning via Quantization). We turn that failure into a measuring tool. Take a fine-tuned model and round its weights onto the base model's own 4-bit grid. Fine-tuning changes too small to cross a bin boundary disappear, and the larger ones remain. The question is whether gains on a benchmark that leaked into fine-tuning vanish faster than gains on matched questions that never leaked.
The answer matters for three reasons. Contamination detectors that read the model's outputs can be defeated by paraphrasing the test before training on it (Evading Data Contamination Detection for Language Models is (too) Easy). Dynamic benchmarks only cover symbolic tasks. And the literature disagrees about whether 4-bit quantization damages reasoning by 1–3% or by more than 30%. A weight-space probe that needs only the base and the fine-tuned checkpoints is a different kind of evidence, and it holds up if the cheaper alternatives below fail.
Core Claim
Our central hypothesis concerns contamination injected during fine-tuning, including paraphrased contamination. We expect its benchmark gain to sit in smaller-than-one-bin weight changes more often than genuine, transferable fine-tuning gains do. If so, rounding to the base model's grid will remove a larger share of the contaminated gain than of the clean gain.
We also expect contamination present in pre-training to behave the opposite way. Because it is part of the base weights themselves, it should survive ordinary quantization better than skill-dependent answers. That would help explain why static benchmarks and dynamic ones respond so differently to quantization.
Both halves of the claim can be falsified, and we treat a null result as informative.
Where This Sits
Work that supplies the mechanism. Catastrophic Failure of LLM Unlearning via Quantization documents the "rounding trap." Retained forgotten knowledge rose from 21% to 83% (MUSE-NEWS) and from 13% to 89% (MUSE-BOOKS) after 4-bit quantization, across three standard quantizers (round-to-nearest, GPTQ and AWQ). That paper studies deliberate unlearning of private or copyrighted text. It never asks about test-set leakage, and it treats rounding as a bug rather than a probe. We reuse its mechanism in the opposite direction. The weight-anchoring theory in Why Fine-Tuning Struggles with Forgetting in Machine Unlearning? explains why fine-tuned solutions stay close to the base weights. Do Unlearning Methods Remove Information from Language Model Weights? recovered 88% of suppressed performance by retraining on related facts. Together these motivate the asymmetry between pre-training and fine-tuning.
Work that motivates the contamination side. Detecting Data Contamination in LLMs via In-Context Learning (called CoDeC) argues that memorized items sit in narrow minima that are easily disturbed. CAP and RADAR detect memorization through two different routes: CAP checks whether answers stay consistent under small rewordings, and RADAR reads internal signals of the model. All three perturb inputs or read activations. None perturbs weights in a way tied to the training history.
Work that constrains the design. The evasion paper shows that paraphrase-trained contamination drives detection to about 1% true-positive rate at a 1% false-positive rate while keeping up to a 15% gain. Any new probe must be tested against exactly that attack. The tension between BeyondBench, which reports a 1–3% drop from 16 to 4 bits on generated problems, and Quantization Meets Reasoning, which reports a 32.39% drop for Llama-3-1B under GPTQ on MATH, is the open question our pre-training arm addresses. Sensitivity of Small Language Models to Fine-tuning Data Contamination reports that more capable small models absorb fine-tuning corruption more readily. That finding fixes our model choice: Phi4-Mini, Qwen2.5-3B and the Llama-3.2 family, where contamination is most likely to stick.
Research Program
The probe. Let $W_0$ be the base weights and $W_1$ the fine-tuned weights. Let $Q_{W_0}(\cdot)$ round a matrix to the 4-bit grid whose per-group scales and offsets were fitted to $W_0$. The probed model is $Q_{W_0}(W_1)$. It is compared against the reference $Q_{W_0}(W_0)$, so that ordinary quantization error cancels.
The controlled contamination study. Each benchmark (GSM8K, MATH-500 and an MMLU subset) is split at random into a leaked half and a clean half. The leaked half is mixed into OpenOrca fine-tuning data in two forms: verbatim copies, and paraphrases produced as in the evasion paper's paraphrase-training attack (Evasive Augmentation Learning). Contamination rates and epochs are varied. Fine-tuning gain on the clean half measures transfer. The extra gain on the leaked half measures memorization.
The headline quantity. We measure the fraction of each gain that survives base-grid rounding, computed per half and per item. For each item, we record whether the fine-tuned model's correct answer survives rounding. Combined with the drop in log-likelihood, that gives a sample-level contamination score.
Lower bit widths. We extend to 3 and 2 bits using QuIP, with the rotation and grid fitted to the base model. This tests whether the effect strengthens as bins widen.
The pre-training arm. We train small models of about 1.7B parameters on FineWeb-Edu, with or without injected benchmark copies, following that paper's ablation scale. We then quantize each model with its own grid and measure accuracy loss. The comparison is between injected items, clean items, and DyVal/BeyondBench generated problems.
Baselines
The naive reference is the full-precision gap between leaked-half and clean-half gains. It is the ground truth in our controlled setting, but it is unavailable to a real auditor, who has no clean half. A second naive reference is ordinary per-model 4-bit GPTQ applied to the fine-tuned model. Because GPTQ refits its grid to $W_1$, it does not specifically remove small changes. If GPTQ separates the two halves as well as base-grid rounding does, the base-grid idea adds nothing.
The strongest existing detectors are CoDeC, CAP's consistency ratio, and Min-K% probability. We run all of them on the same fine-tuned checkpoints and the same items, with thresholds tuned on the same held-out calibration models. Reported numbers are as follows. CoDeC reports dataset-level AUC of 99.9% and RADAR reports 93.0% overall detection accuracy, both on their own settings. The evasion paper reports about 1% true-positive rate at 1% false-positive rate for standard detectors under paraphrase training. We found no reported result on Lacuna for CoDeC or CAP under paraphrase-trained contamination.
Beating the baselines means one of two outcomes. Either our probe achieves higher sample-level true-positive rate at 1% false-positive rate than every detector on the paraphrase-contaminated models, or its dataset-level AUC matches CoDeC's while it uses only weights and no tuned prompts.
Ablations that isolate the ingredient test what actually does the work:
- Gaussian weight noise matched in norm to the rounding perturbation.
- Shrinking the full update $W_1 - W_0$ by a constant factor, with the size of the removed change matched.
- Zeroing every update element smaller than half a bin width, which keeps the "small change" criterion but drops alignment to the grid.
These ablations get the same number of evaluation calls as the probe.
Why not just…?
Why not just perturb the weights with random noise? This is the most serious threat. CoDeC's narrow-minimum picture predicts that any weight perturbation might hurt memorized items more. If matched-norm noise separates the halves as well as base-grid rounding, we will report that the useful signal is generic sharpness and that the rounding trap is not needed. Our reason to expect a difference is that rounding is not isotropic. It deletes precisely the changes made since the base model and leaves the base intact. That should separate memorization added during fine-tuning from the base model's own sharp regions, which random noise cannot do.
Why not just shrink the fine-tuning update, or drop small entries? Interpolating back toward the base model is cheaper and needs no quantizer. The magnitude-threshold ablation tests this directly. If thresholding matches rounding, the scientific claim, that contamination lives in small weight changes, still stands, and quantization is simply a natural instance of it. We would then present the threshold as the tool and say so.
Why not just use dynamic benchmarks? DyVal and BeyondBench are uncontaminated by construction, but they cover only symbolic tasks. They cannot audit MMLU or GSM8K-style claims. We use them as the clean reference in the pre-training arm, not as a replacement for the probe.
Why not just use CAP or CoDeC? Both act on inputs. Paraphrase training is designed so that answers stay consistent under rewording, which may neutralize CAP. Head-to-head runs on the same paraphrase-contaminated checkpoints will show whether this is true.
Decisive Checks
The idea succeeds if the fraction of gain surviving rounding is clearly lower on the leaked half than on the clean half. That gap must hold for both verbatim and paraphrased contamination, across at least three model families. The probe must also beat the noise and shrinkage ablations at matched perturbation size.
The idea fails in either of two ways:
- If the surviving fractions are equal, genuine fine-tuning gains are just as sub-bin as memorized ones, and the probe has no basis.
- If the magnitude threshold matches base-grid rounding, the grid itself contributes nothing beyond removing small changes.
In the pre-training arm, the hypothesis predicts that injected items lose less accuracy under 4-bit and 2-bit quantization than clean static items. It also predicts that the 4-bit accuracy drop on clean static items falls between BeyondBench's 1–3% and the 30%+ drop reported in Quantization Meets Reasoning. That pattern would attribute part of the published disagreement to memorized items on the static benchmarks. If injected and clean items degrade alike, the disagreement comes from the task type rather than from contamination.
References
- Catastrophic Failure of LLM Unlearning via Quantization
- Evading Data Contamination Detection for Language Models is (too) Easy
- Why Fine-Tuning Struggles with Forgetting in Machine Unlearning? Theoretical Insights and a Remedial Approach
- Do Unlearning Methods Remove Information from Language Model Weights?
- Detecting Data Contamination in LLMs via In-Context Learning
- CAP: Data Contamination Detection via Consistency Amplification
- RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation
- BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
- Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning
- Sensitivity of Small Language Models to Fine-tuning Data Contamination
- QuIP: 2-Bit Quantization of Large Language Models With Guarantees
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale