Drag Immunity: Detecting Memorized Benchmark Answers by How Models Recover from Injected Reasoning Errors

When a language model genuinely works through a math problem step by step, a wrong number planted early in its reasoning should hurt it. The error sits in the text the model reads as it writes each next step, and later steps inherit it. When a model has instead memorized the problem and its answer from training data that leaked from a benchmark, the planted error should barely matter: it can jump to the stored answer whatever the intermediate text says. This proposal turns that difference into a per-question contamination test. We corrupt a model's partial solution, measure how much its final answer changes, and check whether it reaches the right answer without ever noticing or fixing the error. A model that keeps reaching correct answers this way, which we call drag immunity, is showing a behavioral fingerprint of memorization.

The test matters now because the usual ways of detecting contamination are breaking down. Most detectors look for unusually low "surprise" (high log-probability) on test text the model may have seen. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that reinforcement learning (RL) after contaminated fine-tuning erases this footprint. Contaminated chain-of-thought training inflates pass@1 by 11.7% on AIME, AMC and GPQA Diamond, yet all ten detectors tested fall to 50–58% AUROC (area under the ROC curve, where 50% is chance). RL leaves the inflated score in place and removes the evidence. Meanwhile cheap low-rank adaptation (LoRA, which trains a small low-rank add-on to frozen weights) has made RL-trained reasoning models nearly free to produce. Tina reaches 43.33% AIME24 Pass@1 for about $9 with LoRA plus GRPO, a group-based RL method, and never audits its benchmarks. We need a detector that probes behavior rather than token probabilities, and LoRA makes it affordable to build the many controlled contaminated models needed to validate one.

Core Claim

Sensitivity to injected intermediate errors is a membership signal that survives RL cloaking. For a question $q$ and a step depth $j$, define drag immunity as

$$D_j(q) = \frac{P(\text{correct} \mid q, \tilde{s}_{1:j})}{P(\text{correct} \mid q, s_{1:j})},$$

where $s_{1:j}$ is the model's own first $j$ reasoning steps, $\tilde{s}_{1:j}$ is the same prefix with one planted numerical or logical error, and both probabilities are estimated by sampling continuations. For genuinely reasoned items, $D_j$ should fall below one, and it should fall further when the error is planted late, because less remaining computation is available to repair it. For memorized items, $D_j$ should stay near one at every depth, and the recoveries should be unfaithful: the model reaches the answer without acknowledging the error.

We expect the signal to survive RL for a specific reason. The fragility paper attributes cloaking to clipped policy updates that shrink the log-probability gap between seen and unseen items. That mechanism acts on how likely the text is, not on whether the answer is retrieved independently of the reasoning. Tina's "rapid reasoning format adaptation" hypothesis, that LoRA RL mainly reshapes output format rather than stored knowledge, predicts that the retrieval route persists under LoRA RL. That would leave drag immunity intact even after likelihood signals are gone.

Relation to Prior Work

Two papers motivate the signal. Faithful and Unfaithful Error Recovery in Chain of Thought injects calculation and copying errors into GSM8K-style transcripts. It finds models often reach correct answers without repairing the error, but it never asks whether memorization explains this. We treat that unexplained recovery as the detector. Contextual Drag shows that flawed context costs reasoning models 10–20% accuracy even when the model is warned. It explicitly leaves open whether memorized problems are immune. We answer that question and turn it into a test.

Two other papers constrain the design. The fragility paper rules out likelihood-based detectors after RL. Faithful-recovery work also reports that strong models such as GPT-4 recover faithfully from large errors. Recovery alone therefore cannot be the signal, which is why our statistic conditions on faithfulness and on the depth profile rather than on raw recovery.

The method draws on several sources. Step segmentation and first-error labeling follow the conventions of ProcessBench, whose finding that 51.8% of correct Omni-MATH solutions contain process errors warns that correct answers over bad steps also occur without contamination. Cheap LoRA GRPO training follows Tina. Embedding-based decontamination of the training data follows Platypus.

One Rank at a Time is the only file paper joining error cascades with LoRA. Its cascades are between rank components in linear regression, not between reasoning steps, so it does not overlap with this work.

Research Program

The first stage builds ground truth. Starting from DeepSeek-R1-Distill-Qwen-1.5B (Tina's base), we split AIME, AMC23 and MATH500 items at random into members and non-members. We run LoRA supervised fine-tuning on reference chain-of-thought solutions for members only. We then apply clean LoRA GRPO with Tina's recipe on decontaminated data, with checkpoints at zero, one and several RL epochs. We repeat with full-parameter training at matched data, and vary the contamination dose (one versus several exposures) and the form of exposure (answer only, versus full solution). LoRA keeps the full grid at tens of dollars per model, so we can train enough models to report variance across seeds and splits.

The second stage builds the probe. For each item we sample the model's own solution, segment it into steps, and plant errors at several relative depths. The error types are a changed intermediate number, a wrong operation, and a flawed whole draft in the style of contextual drag. We sample continuations from each corrupted prefix and from each clean prefix. A held-out critic model labels whether each continuation acknowledges and repairs the error. The final score combines the mean of $D_j$, its slope across depth, and the unfaithful-recovery rate, with a weighting chosen once on a single calibration model and then frozen.

The third stage audits released LoRA reasoning checkpoints such as Tina's. Ground truth is unavailable there, so the audit is descriptive. We report how drag-immunity scores are distributed on older versus newer competition items rather than claiming any item is contaminated.

Baselines

The naive reference is chance (50% AUROC), together with the raw accuracy gap between members and non-members, which shows how much contamination inflated scores in each model.

The strongest existing competitors are the likelihood detectors Loss, Min-K%++ and LiRA. LiRA is the reference point: the fragility paper reports 89.1% AUROC after contaminated SFT, falling about 14% after one GRPO epoch, with all detectors at 50–58% for chain-of-thought contamination of reasoning models. We run them on identical member/non-member splits and checkpoints, with reference models for LiRA trained under the same LoRA recipe. Beating them means holding at least 70% AUROC after RL where they sit near chance, while remaining competitive before RL.

A question-perturbation baseline rewrites each item with altered numbers and a recomputed answer, then scores the drop from original to variant. It gets the same sampling budget as our probe, measured in total generated tokens per item.

Two ablations isolate the key ingredient. The first plants no error and simply forces the model's own clean prefix. This tests whether conditioning on a prefix alone, rather than on a corrupted one, separates the two groups. The second drops the faithfulness label and keeps only the accuracy ratio, which tests whether the critic adds anything over raw recovery.

Why not just…?

Why not just use perturbed question variants, the standard contamination check? This is the real threat. It may detect contamination as well as our probe on easy arithmetic sets. Its weakness is that it needs a new problem with a verified new answer and matched difficulty for every item. That is costly and error-prone for AIME-level questions, and any difficulty shift is confounded with memorization. Our probe leaves the question and the gold answer untouched. We run both under the same token budget. If perturbation matches us after RL, we report that honestly and test whether combining the two signals helps. We expect the two to diverge most on hard items, where faithful variants are scarce.

Why not just recalibrate likelihood detectors for RL-trained models? The fragility paper traces the collapse to clipping that compresses the very gap these detectors measure, so recalibration has little left to work with. We include a recalibrated LiRA trained on post-RL reference models. If it recovers, our case rests on the probe's need for less access, since it uses no log-probabilities and works through a sampling API.

Why not just evaluate on contamination-resistant benchmarks like LiveBench or BeyondBench? For new evaluations, that is the right choice, and nothing here argues against it. It cannot audit scores already reported on static sets, such as Tina's AIME24 figure, or tell which items in a fixed benchmark are compromised. Contamination-resistant benchmarks avoid the problem going forward, while our probe diagnoses contamination in benchmarks already in use.

Decisive Checks

The central test is AUROC across RL epochs on the controlled models. The claim fails if drag-immunity AUROC collapses toward chance alongside LiRA after GRPO. A second check asks whether the depth profile separates the groups as predicted, flat for members and falling for non-members, rather than all items simply recovering. Contextual Drag's finding that 1.5–7B models lose roughly half their accuracy under flawed context implies non-members should show $D_j$ well below one. If they do not, the probe has no room to work at this model scale and must move to a larger base.

A third check compares LoRA and full-parameter training. If LoRA RL preserves drag immunity better than full fine-tuning, that supports Tina's format-adaptation hypothesis. It would also imply that cheap LoRA reasoning models leave more auditable traces than their full-parameter counterparts.