Contaminated Answers Survive Low-Rank Compression: A Two-Sided Weight Probe for Hidden Benchmark Leakage

When a language model has seen a benchmark's test questions during training, its score on that benchmark partly measures recall instead of reasoning. The usual way to catch this is to look for a loss gap, meaning that seen questions get lower loss than unseen ones. Recent work shows the gap can be erased. A single epoch of reinforcement learning on clean data after contamination drives standard detectors toward chance, and the inflated score stays. We propose a different signal, drawn from model compression. Memorized answers appear to run through a few dominant internal directions, while genuine multi-step reasoning uses many. If so, the two kinds of answers should respond in opposite ways to two cheap weight edits. Keeping only the top directions of each weight matrix, as low-rank compression does, should preserve memorized answers and damage reasoning. Removing those top directions should do the reverse.

The core claim is that this asymmetry between "keep the top" and "remove the top" identifies contaminated benchmark items one by one. Because it depends on how the model routes computation and not on loss values, it should still work after reinforcement learning has hidden the loss gap. The same measurements also answer a practical question for anyone who ships compressed models: when a contaminated model is shrunk, does its inflated score shrink with it, or does compression keep the cheating and discard the reasoning?

Why this is the right intersection

Three findings motivate the proposal. A Narrowing Geometry in Contaminated Reasoning shows that on leaked math items, activations in MLP projection layers concentrate in a low-dimensional subspace. Projecting out the dominant eigenvectors of mlp.up_proj, a method it calls Top-Subspace Suppression (TSS), restores genuine reasoning on about half of leaked samples and costs under 6% on clean data. That paper uses suppression only as a repair tool. It never asks what happens under the opposite edit, which is exactly what low-rank compression performs. On The Fragility of Benchmark Contamination Detection in Reasoning Models motivates the problem: after SFT contamination, LiRA (a membership test that compares the model's loss against reference models) reaches 89.1% AUROC, but one epoch of GRPO on clean data cuts it by about 14 points. When reasoning models are contaminated with chain-of-thought data, ten detectors fall to 50–58% AUROC. Reasoning or Memorization? shows that contamination in the wild is real and detectable by prefix probing: Qwen2.5 recovers 65.8% exact-match answers on MATH-500 from 80% prompt prefixes. Its leakage-free RandomCalculation generator gives a clean control.

Two papers constrain the design. Knowledge Distillation as Decontamination? finds that distillation into a smaller classifier mostly filters out contamination. On tweet_sentiment, a 25.66% direct gain shrinks to 3.25% in the student. This warns that some forms of size reduction do act as bottlenecks, and it is limited to classification. We study weight-level compression on generative reasoning, which it explicitly leaves open. Data Contamination: From Memorization to Exploitation shows that memorization and exploitation can come apart, with about 15% memorization alongside under 1% exploitation in BERT-base. Our probe therefore has to measure whether a correct answer survives, not whether text is memorized. Pruning vs Quantization: Which is Better? supplies the compression arms and a caution: 4-bit quantization and pruning damage weights very differently depending on outliers. On OPT-350m, 4-bit quantization gives 15.3 perplexity versus 35.4 at 75% pruning. Quantization is therefore a third, roughly direction-blind perturbation we must include.

Core Claim

For an item $x$ with reference answer $y$, let $p_0(x)$ be the model's probability of producing $y$. Let $p^{\text{keep}}_k(x)$ be the same probability after replacing selected weight matrices with their rank-$k$ truncated SVD. Let $p^{\text{drop}}_k(x)$ be the probability after projecting out the top-$k$ directions instead. The asymmetry score is

$$A(x) = \frac{1}{|K|}\sum_{k \in K} \left[ \log \frac{p^{\text{keep}}_k(x)}{p_0(x)} - \log \frac{p^{\text{drop}}_k(x)}{p_0(x)} \right],$$

where $K$ is a small grid of ranks. Our hypothesis predicts that contaminated items have high $A$: they survive truncation and die under removal. Clean items solved by reasoning should have low or negative $A$. The score needs open weights but no reference models, no training data, and no access to the loss gap that reinforcement learning erases. A forward pass costs about $2|K|$ times one inference per item. That is cheaper than repeated sampling and far cheaper than influence functions.

The idea is close to TSS but not the same. TSS removes directions to repair a model. We pair removal with its mirror image, the edit that low-rank compression makes, and read the difference between the two as a per-item test. If memorized items do not survive truncation better than reasoned items, the claim fails. Experiment one tests exactly that.

Research Program

The program first builds a controlled testbed with known membership, in the style of the Fragility paper. We contaminate Qwen2.5-7B-Instruct by SFT on a random half of AIME, AMC, OlympiadBench and MATH-500 items, then run one epoch of GRPO on clean data. We also build a Stage II variant by fine-tuning DeepSeek-R1-Distill-Llama-8B on benchmark chain-of-thought traces. At smaller scale, we contaminate Qwen2.5-1.5B on GSM8K and on MSCalc, a synthetic rational-arithmetic set from the Narrowing Geometry paper. The held-out half of each benchmark serves as non-members of matched difficulty.

Next we decide where the directions come from, to avoid circularity. The main arm uses weight SVD, which needs no data. A second arm uses top eigenvectors of activation covariance on a generic calibration text set that contains no benchmark data. A diagnostic arm uses the item's own activations, as TSS does, to measure how much is lost by not using per-item information. We apply edits to mlp.up_proj first, since that is where TSS worked, then extend to attention projections, which that paper leaves unexamined.

Finally we run the compression question directly. On each contaminated model we apply INT4 and INT8 uniform quantization, 50% and 75% magnitude pruning, and rank-truncation at matched memory budgets. We then measure how much of the contamination-driven inflation remains. The inflation is the gap between member and non-member accuracy. Clean reasoning is measured on DyVal tasks at difficulty tiers D1–D4 and on RandomCalculation. This produces a map of which compression methods launder contamination and which strip it.

Baselines

The do-nothing reference is raw per-item correctness or answer log-probability $p_0(x)$ used as a membership score. Contaminated items are usually answered more confidently, so any proposed signal has to beat plain confidence. All detectors use the same contaminated checkpoints, the same member and non-member splits, and the same items.

The strongest existing competitor is LiRA as run in On The Fragility of Benchmark Contamination Detection in Reasoning Models. That paper reports 89.1% AUROC after SFT contamination, a drop of about 14 points after one GRPO epoch, and 50–58% AUROC across ten detectors in the Stage II chain-of-thought setting. We reuse its reference-model training recipe and give it the same compute budget as our probe or more. Beating it means clearly higher AUROC after GRPO and above 58% in Stage II. Matching it before GRPO is acceptable.

The strongest black-box competitor is CDD from Generalization or Memorization. CDD flags contamination when many sampled outputs sit at near-zero edit distance from one another. That paper reports a 21.8–30.2% relative gain in accuracy and AUC over n-gram and LLM baselines on DETCON. We run CDD with a sample count chosen so its forward-pass budget matches $2|K|$. We also run it at its original budget, and we include TED filtering for the inflation-estimation task. A loss-based detector and partial-prompt answer accuracy from Reasoning or Memorization? round out the set. For partial-prompt accuracy, the cited Qwen2.5 MATH-500 figure is 65.8% exact match at an 80% prefix.

Three ablations isolate the key ingredient. Drop-only scoring is essentially TSS used as a detector. Keep-only scoring is rank truncation alone. Norm-matched random-direction removal tests direction specificity. The claim needs the two-sided score to beat both one-sided scores and to clearly beat random directions.

Why not just…?

Why not just compare scores on the benchmark against fresh or rephrased versions, as DyVal or HumanEval_R do? This captures aggregate inflation well, and we use it to define ground-truth inflation. It cannot say which items leaked, and it needs a matched fresh set for every benchmark. DyVal shows that difficulty shifts under regeneration: GPT-3.5-Turbo drops 23% from D1 to D4. That mixes harder problems with contamination. Our probe works per item on the original benchmark. If per-item detection turns out not to matter for estimating corrected scores, this alternative wins on simplicity, and our inflation-estimation experiment will show whether it does.

Why not just use one-sided top-subspace suppression as the detector? This is the most serious threat, because the Narrowing Geometry paper already shows that suppression separates leaked from clean behavior. Suppression alone also hurts hard clean items that use many dimensions, which confuses "fragile" with "memorized." The keep-arm is meant to remove this confusion. If drop-only scoring matches the two-sided score in AUROC, we drop the claim of a new detector and report the work as an extension of TSS to reinforcement-learning-cloaked and compressed settings.

Why not just add random weight noise and measure fragility? If memorized answers are simply more or less robust to any perturbation, a direction-blind probe would suffice. The norm-matched random-direction arm and the INT4 arm test this directly. A result with no gap between them and our score would show the geometry adds nothing, and we would say so.

Why not just distill into a smaller model and trust the bottleneck? Knowledge Distillation as Decontamination? found filtering only for classification with soft labels. In that study, soft labels transferred more contamination than hard labels. Distillation is also expensive and produces a different model, not a verdict on the original. We report distillation only as a reference point in the compression map.

Proposed Experiments

The first decisive check is whether memorized items survive rank truncation better than reasoned items. On the 1.5B GSM8K and MSCalc models, members must show reliably higher $A(x)$ than non-members of matched difficulty, and the effect must exceed the random-direction arm. If truncation hurts members and non-members equally, the project stops here.

The second check is the detection test after reinforcement learning. We report AUROC of $A(x)$ against all baselines on the GRPO-cloaked 7B model and on the Stage II R1-Distill model. The target is to stay well above the 50–58% band reported for existing detectors.

The third check is an in-the-wild test without planted contamination. We score MATH-500 items on public Qwen2.5 checkpoints and test whether $A(x)$ correlates with per-item partial-prompt recovery. We expect near-zero $A$ on RandomCalculation items, which cannot have leaked.

The fourth check is the compression map. For INT4, INT8, magnitude pruning and rank truncation at matched size, we report retained contamination inflation alongside DyVal and RandomCalculation accuracy. The prediction is that rank truncation keeps the most inflation per unit of clean reasoning lost. If quantization strips inflation while truncation preserves it, that is a direct warning: compressed-model leaderboards built on static benchmarks can overstate how well reasoning survives compression.