Unlearning as a Measuring Instrument: Recovering the Clean Score of a Contaminated Model

When a language model has seen a benchmark's questions and answers during training, its score on that benchmark mixes two things: what it can actually solve and what it has memorized. The usual responses are to throw the benchmark away or to build fresh ones. This proposal takes a different route. It asks whether targeted forgetting can remove only the memorized part of a model's knowledge of a specific benchmark, so that the model's new score estimates what it would have scored had it never seen the test. Targeted forgetting here means a short fine-tuning procedure that erases specific learned content while keeping everything else.

The key move is to judge forgetting methods by calibration rather than by how much they erase. In a controlled setting we train "twin" models that are identical except that one saw the benchmark and one did not. A good decontamination method should bring the contaminated twin's score down to the clean twin's score, item by item. It should not push the score lower than that, which would mean it erased real skill, and it should not leave it higher, which would mean memorization remains. It should also hold that score when the model is attacked with decoding tricks known to bring supposedly forgotten content back. If this works, a lab holding a suspect checkpoint could get a contamination-corrected score on a static benchmark for a few GPU-hours, instead of retraining or abandoning the benchmark.

Core Claim

Existing forgetting methods pursue the wrong target for this use. Safety unlearning tries to drive performance on the forget set toward zero. Decontamination needs performance to land on a counterfactual value: what the model would score on the leaked items using genuine skill alone.

We propose contrastive trace forgetting. A forget signal targets the model's behavior that is specific to the exact item. A retain signal targets the model's own reasoning on perturbed twins of the same items, meaning the same problem structure with changed numbers or entities. A stopping rule tells the procedure when to halt without access to the clean model. The stopping rule is the partial-prompt completion audit: the model is shown only the first 40%, 60%, or 80% of a problem. Training halts when the model's accuracy from truncated prompts falls to the level expected of an uncontaminated model, not to zero.

The claim is testable and falsifiable. On controlled twins, the method's per-item agreement with the clean twin should exceed that of inference-time subspace suppression, passive continued pretraining at matched compute, and scoring on paraphrased variants. It should also survive decoding that strips out the reasoning trace.

Where This Sits

Three lines of work motivate the proposal.

How Much Can We Forget about Data Contamination? injects benchmark items into FineWeb-Edu pretraining at controlled repetition rates and shows that models forget them passively under continued pretraining. Continuing OLMo-7B for 13% of its schedule removed 87% of the contamination gain. We borrow its twin-model injection protocol as ground truth. We replace passive forgetting with an active procedure that is orders of magnitude cheaper. We also test the reasoning traces and paraphrased contamination that the paper leaves open.

Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination shows that Qwen2.5-Math-7B answers nearly 40% of MATH-500 from the first 40% of the problem, while Llama3.1-8B scores near zero. That partial-prompt audit supplies our stopping signal and one of our leakage metrics.

A Narrowing Geometry in Contaminated Reasoning finds that contaminated reasoning collapses into a low-rank activation subspace. Its Top-Subspace Suppression (TSS) restores base-model-like behavior on about half of leaked GSM8K and MSCalc samples. This is the closest existing method. It differs from ours in three ways: it clamps activations at inference rather than changing weights, it is limited to linear MLP projections, and it was not audited under trace-suppressing decoding.

Two unlearning papers supply our method and also constrain it.

Rethinking Unlearning for Large Reasoning Models (also Reasoning Model Unlearning) introduces R²MU. R²MU maps hidden states of forget-set reasoning traces to random vectors while regularizing on LIMO reasoning data. It cuts trace leakage on WMDP from 19.71% to 1.02% and keeps roughly 84% on MATH-500. Without the reasoning regularizer, MATH-500 accuracy collapses to 0%. That collapse is a warning for us. Contaminated items share their skills with clean math, so naive misdirection on MATH items is likely to destroy the very skill whose level we are trying to measure. This is why our retain set is built from the same items with surface details changed, not from a generic corpus.

R-TOFU: Unlearning in Large Reasoning Models shows that answers believed forgotten come back under ZeroThink and LessThink decoding, which suppress or truncate the model's reasoning trace before it answers. We adopt these decoding attacks as a required audit.

Two procedural benchmarks supply clean, structure-preserving variants for building perturbed twins and for checking against real contamination: DyVal and RandomCalculation.

Research Program

The method has three parts.

The forget set is built from the contaminated model's own outputs on leaked items. These include the exact-item solution trace and the completions the model produces from truncated prompts, since truncated-prompt completions are the clearest sign of memorized retrieval.

The retain set comes from perturbed twins of each leaked item. For GSM8K and MATH this means the same solution structure with re-sampled constants, checked by a solver. For MSCalc and DyVal-style items this means regenerated instances from the same template. On these twins we keep the model's hidden states and output distribution close to the pre-intervention model's.

The objective follows R²MU's form, with $\lambda$ weighting the retain term and a small LIMO term to protect general reasoning:

$$ \mathcal{L} = \mathbb{E}_{x \in F}\,\lVert h_\ell(x) - c\,u \rVert^2 + \lambda\, \mathbb{E}_{x' \in T}\,\lVert h_\ell(x') - h^{0}_\ell(x') \rVert^2 . $$

The symbols are as follows. $F$ is the forget set and $T$ is the perturbed-twin retain set. $h_\ell$ is the layer-$\ell$ hidden state of the model being edited and $h^0_\ell$ is that of the frozen original. $u$ is a fixed random unit vector and $c$ is a scale constant.

Training is checked every few steps against the partial-prompt audit and stops when truncated-prompt accuracy on leaked items matches truncated-prompt accuracy on held-out items of the same benchmark. Held-out items stand in for the clean twin. This works because a model's ability to finish a problem it has only partly read should be about the same for items it has seen and items it has not, unless it has memorized the seen ones.

The analysis side asks two questions. First, which kinds of contamination this procedure can and cannot reverse: verbatim versus paraphrased injection, answer-only versus full-trace exposure, and low versus high repetition. Second, whether the procedure removes the low-rank signature from A Narrowing Geometry in Contaminated Reasoning in weight space, or only hides it. We measure that signature with its Eigenspectrum Concentration Index, a measure of how concentrated the model's internal computation becomes in a few directions.

Baselines

The naive reference point is the contaminated model scored as-is, paired with the clean twin as the oracle target. Every method is reported as a distance between those two numbers. A method that does not close most of the gap between them is not useful.

The strongest existing method is TSS from A Narrowing Geometry in Contaminated Reasoning. It reports an 11.03% absolute improvement in consistency with the uncontaminated base model over LNE-Blocking, recovery of genuine reasoning on approximately half of leaked samples, and under 6% degradation on clean data, on GSM8K and MSCalc. We will run TSS with the authors' layer choice (mlp.up_proj) and tune its suppression rank on the same held-out validation items our method uses. We will report it on the same twins with the same consistency metric. Beating it means three things: higher item-level consistency with the clean twin on GSM8K and MSCalc, no larger clean-data drop, and a score that holds under ZeroThink decoding where TSS's does not.

The compute-matched passive baseline is continued pretraining on clean FineWeb-Edu, following How Much Can We Forget about Data Contamination?. Its reported result, an 87% reduction in contamination gain for OLMo-7B after an extra 13% of the schedule, comes from a far larger budget. We will report two configurations: one at the same GPU-hours as our method, and one at the paper's 13% budget where feasible. That shows where the cost curves cross.

The answer-level unlearning baselines are gradient ascent on the leaked question–answer pairs and R²MU unchanged. Both are given the same forget items, the same LIMO regularization, the same tuning sweep over learning rate and $\lambda$, and the same stopping rule. This isolates whether our gain comes from the perturbed-twin retain set or from the stopping rule.

We run two ablations. One replaces the perturbed-twin retain set with LIMO alone, which isolates the key ingredient. The other replaces the audit-based stopping rule with a fixed step count, which tests whether calibration depends on knowing when to stop.

Why not just…?

Why not just score the model on perturbed or paraphrased variants and skip editing it? This is the most serious threat, because it is cheap and needs no access to the model's weights. The survey reports that models lose 39.4% on average when moved from HumanEval to EvoEval (Benchmark Data Contamination of Large Language Models: A Survey). That drop mixes removed contamination with changes in difficulty, and variants can themselves be soft-contaminated (How Much Can We Forget about Data Contamination?). We treat this as a full competing estimator. On the twins, we compare its error against the clean twin's score with our method's error. If variant scoring is as accurate, we will report that decontaminating by editing the model is unnecessary for score correction. In that case, its remaining value would be producing an edited model that can serve as a clean starting point for later experiments, such as reinforcement learning studies.

Why not just move to dynamic benchmarks like DyVal or LiveBench? These benchmarks avoid contamination for new evaluations. They cannot correct historical results on MATH, GSM8K, or MMLU, and they are confined to domains where answers can be generated and checked automatically (DyVal; LiveBench). We use them as validation tools, not as competitors.

Why not just apply TSS at inference? TSS is cheap and was not designed around a counterfactual target. It does, however, recover only about half of leaked samples and has not been tested against trace-suppressing decoding. The head-to-head comparison with ZeroThink and LessThink audits settles whether a change to the weights is needed.

Why not just continue pretraining? The cost reported by Bordt et al. is billions of tokens, and their finding that spaced repetition makes contamination harder to forget suggests that passive forgetting is weakest exactly where contamination is most realistic. The compute-matched run shows this directly.

Proposed Experiments

The controlled study uses three models: Llama3.1-8B, chosen because its partial-prompt MATH-500 score is near zero and it is therefore plausibly clean; Qwen2.5-1.5B; and DeepSeek-R1-Distill-Llama-8B, a reasoning model that writes out its chain of thought. Each model receives a continued-pretraining mixture with and without injected GSM8K, MATH-500, and MSCalc items. Injection varies along three axes: repetition (4x to 64x), form (verbatim versus paraphrased), and exposure (answer only versus full solution trace). This yields matched twins for every condition.

The primary metric is the absolute gap between the decontaminated score and the clean-twin score, together with item-level agreement with the clean twin. Secondary metrics are:

  • partial-prompt accuracy at 40%, 60%, and 80% truncation;
  • accuracy under ZeroThink and LessThink decoding;
  • utility on AIME 2024, GPQA Diamond, and RandomCalculation;
  • the Eigenspectrum Concentration Index before and after intervention;
  • GPU-hours.

The decisive checks are three. First, our method must beat TSS and variant-scoring on calibration error. Second, its corrected score must not rise again under ZeroThink. Third, the ablation without perturbed twins must reproduce R²MU-style over-forgetting, with scores falling below the clean twin. If the first check fails against variant-scoring, the central claim fails.

As a final transfer test, we apply the method to Qwen2.5-Math-7B, which has no twin. We check whether its corrected MATH-500 score drops toward its RandomCalculation-calibrated level while its truncated-prompt accuracy falls from nearly 40% toward the near-zero level reported for Llama3.1-8B.