Does the Optimizer Decide Whether Benchmark Contamination Can Be Caught?

When a language model has seen test questions during training, its benchmark score goes up for the wrong reason. This problem is called benchmark contamination. Auditors try to catch it by looking for traces the leaked items leave behind, such as unusually low loss, overconfident rare tokens, or brittleness under paraphrase. Nearly all of this work treats those traces as a property of the data: how many test items leaked, and how often.

We argue that the traces also depend on how the model learned the leaked items. Adaptive optimizers such as Adam and AdamW give each weight its own step size, rescaled by a running record of its past gradients. They are known to change which solution a network reaches: they lean toward rare features, and their updates are driven mainly by the sign of the gradient. If the optimizer changes how a leaked item is stored, it should also change how easily the item can be detected.

The proposal is a controlled study that measures this dependence directly. We inject benchmark items at known rates. We then train under a grid of optimizer settings. For each setting we measure two quantities: how much the leak inflates the score, and how detectable the leak is to current auditing methods. The practical question is whether some ordinary training configurations are "stealthy," meaning they buy the same score inflation with much less detectable evidence.

If they are, audit sample sizes calibrated on one training recipe will be wrong for another. Auditors would then need to know, or bound, the optimizer that was used.

Core Claim

The central quantity is an exchange rate between inflation and detectability. When Is Benchmark Contamination Detectable? models an audited model's behavior as a mixture of clean and contaminated behavior. It shows that audit power is governed by the product $\alpha\rho\sqrt{m}$, where the terms are:

  • $\alpha$, the fraction of items that are contaminated;
  • $\rho$, the behavioral separability between contaminated and clean items, measured as a $\chi^2$-divergence;
  • $m$, the number of items audited.

Holding power fixed, the number of audit items needed scales roughly as

$$m^* \propto \frac{1}{(\alpha\rho)^2}.$$

This scaling follows directly from requiring $\alpha\rho\sqrt{m}$ to reach a fixed threshold. That paper also shows the textbook Gaussian version of this formula fails at small $m$. We therefore use its simulation-based planner for actual sample sizes.

That work treats $\rho$ as an empirical property of each injection. Our claim is that $\rho$ is substantially controlled by the optimizer, and that it can move in ways not explained by how strongly the items were learned.

Concretely, we define the stealth of a training configuration as the score inflation it produces per unit of $\rho$, measured at matched inflation. We predict that stealth varies across settings that practitioners use routinely: SGD versus Adam, the second-moment memory $\beta_2$, decoupled weight decay, and learning rate. We also predict the variation is large enough to change audit sample sizes by factors that matter.

There are three mechanistic reasons to expect this.

First, The Marginal Value of Adaptive Gradient Methods in Machine Learning argues that per-coordinate rescaling pushes learning toward rare, spurious features. Benchmark-specific tokens are exactly such features. Adam should therefore store leaked items as sharp, token-level memorization. Detectors that focus on rare tokens, such as Min-K%++, should see a large likelihood gap under Adam.

Second, SGD's updates are proportional to gradient magnitude, so they are dominated by directions shared across many examples. Under SGD, contamination may be absorbed more as a reusable pattern. This would resemble the chain-of-thought contamination in On The Fragility of Benchmark Contamination Detection in Reasoning Models, where confidence rises on seen and unseen similar questions alike. The result would be inflation that also spills onto paraphrases, with little separability.

Third, Adam Reduces a Unique Form of Sharpness shows that Adam and SGD minimize different curvature measures. This gives a theoretical reason to expect the two to reach different minima even when training on identical data.

Which direction the effect goes is an empirical question. The note commits only to measuring it cleanly, including the null outcome.

Research Program

The program has three parts.

Part one: continued pretraining. We take the SmolLM 360M and 1.7B models used by Memorization Sinks and continue training them on a SlimPajama-style mix. Into this mix we inject GSM8k and MATH test items at several contamination fractions and repetition counts.

We train two kinds of copies. The first kind contains verbatim items. The second kind contains in-distribution variants, meaning paraphrased or renumbered problems, following DICE; these come from the GSM-hard and GSM-i-Syn style variants. Each injection is trained under the following optimizer settings:

  • SGD with momentum;
  • Adam;
  • AdamW at two weight-decay values;
  • AdamW with $\beta_2 \in \{0.95, 0.999\}$;
  • a sign-momentum optimizer.

The sign-momentum optimizer isolates the sign-dominance of Adam's updates reported in Dissecting Adam. Each setting is swept over learning rate.

For every trained model we record two things. The first is inflation: accuracy on the leaked items minus accuracy on held-out items matched in difficulty. The second is the full detector suite, from which we estimate $\rho$.

Part two: separating inflation from learning strength. The key analysis move is to compare configurations at matched inflation rather than at matched steps. Otherwise "Adam is more detectable" could simply mean "Adam learned the items more."

For each configuration we sweep the learning rate and number of exposures to trace a curve of inflation against $\rho$. We then compare these curves, not single points. If all optimizers fall on one curve, optimizer choice matters only through how strongly the items are learned. That is itself a useful negative result, because it would license optimizer-agnostic audit calibration.

Part three: the reinforcement-learning stage. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows the following sequence: contaminated supervised fine-tuning, then clean reinforcement learning with GRPO or RAFT++, collapses likelihood detectors. The paper attributes this to the clipping in the reinforcement-learning objective, which bounds how far each update can move the probability of the sampled answer. It does not examine the optimizer.

Adam's normalization makes even tiny reward gradients produce sizable per-weight steps. This could independently overwrite the item-specific weights that carry the contamination signal.

We reproduce their protocol on AIME, AMC, OlympiadBench, and Minerva Math using a distilled reasoning model in the 1.5–8B range. We then cross three factors:

  • clipping on versus off;
  • Adam versus SGD in the reinforcement-learning stage;
  • carrying over versus resetting the optimizer's moment estimates from fine-tuning.

We measure how much of the detection collapse each factor explains.

Deliverables. The work produces three things:

  • measured $\rho$ and stealth surfaces over optimizer settings;
  • a recalibration of the Shihab et al. planner that takes optimizer family as an input, tested on how well it predicts power for held-out configurations;
  • a decomposition of reinforcement-learning masking into objective-driven and optimizer-driven parts.

Baselines

Optimizer-agnostic calibration (the naive reference). This is the "do nothing" reference. We run the two-stage planner of When Is Benchmark Contamination Detectable? calibrated on a single default recipe, AdamW with $\beta_2=0.95$. We then ask it for the audit size needed to reach 0.80 power on every other configuration.

The original paper reports the planner delivering 0.785–0.92 power against a 0.80 target, and predicting power across held-out channels with $R^2$ up to 0.98. Those channels varied the injection, not the optimizer.

Beating this reference means one of two things. Either the optimizer-agnostic planner misses its power target by clearly more than that reported band on some configurations, or an optimizer-aware version restores the band. If the agnostic planner stays within 0.785–0.92 everywhere, our central claim fails.

Likelihood detectors. LiRA, loss, and Min-K%++ are the standard black-box detectors. The Fragility paper reports LiRA at 89.1% AUROC before reinforcement learning. After GRPO, LiRA drops by about 14% and loss by about 16%. On chain-of-thought contamination, all ten of its detectors average 50–58% AUROC. These are the reference numbers our reproduction must hit before any optimizer comparison is trusted.

Representation detectors. These are the strongest detection competitors. DICE finds the layer whose hidden states shift most and trains a small classifier on it; it reports AUROC near 100 on in-distribution GSM8k contamination. CANARY compares hidden states before and after fine-tuning through sparse autoencoders; it reports AUROC 1.000 at 1% contamination, falling to about 0.83 under low-rank LoRA fine-tuning.

That LoRA drop is the one reported case where the structure of the update, not the data, weakened a strong detector. It motivates asking whether optimizer choice does the same.

Reinforcement-learning-stage detector. For the reinforcement-learning stage we add the self-critique entropy probe of Detecting Data Contamination from Reinforcement Learning Post-training. It reports average AUC 0.70 on Qwen2.5-7B-Instruct and 0.64 on DeepSeek-Math-7B-Instruct, where perplexity and Min-K% sit at 0.42–0.53.

Fairness controls. Every detector sees the same models, the same candidate items, and the same audit budget $m$. Hyperparameters are tuned once on a held-out calibration configuration and then frozen. This way no detector is re-tuned per optimizer.

Ablations. The key ablation compares at matched inflation versus matched training steps. The Part-three factorial isolates clipping from optimizer normalization. Sign-momentum versus Adam separates sign-dominance from magnitude adaptation.

Why not just…?

Why not just attribute any difference to how strongly the items were learned? This is the most serious threat. Adam at a given learning rate may simply memorize more, making it trivially more detectable. The matched-inflation curves exist to settle this. We compare optimizers only where they yield the same inflation.

If the curves coincide, we report that optimizer choice is irrelevant for audit calibration beyond memorization strength. That outcome would still correct the implicit assumption in the Shihab et al. planner by confirming it.

Why not just use a representation detector, which already reaches near-perfect AUROC? The perfect DICE and CANARY numbers come from single training recipes. CANARY's own LoRA result, about 0.83, shows that update structure can erode it. Representation detectors also need access to the model's weights, and CANARY needs the weights from before fine-tuning. Most public audits of released models are black-box.

We include both detectors across the full grid. If they stay near perfect under every optimizer, we will say plainly that optimizer-driven stealth threatens only black-box audits. That is still the regime in which most leaderboard claims are checked.

Why not just read the optimizer's stored gradient statistics as an audit trail? Adam's second-moment estimate is appealing because it keeps a decaying memory of past squared gradients. However, it pools gradients over every example in every batch, so it cannot attribute anything to individual items. With $\beta_2=0.999$, its memory also fades over roughly a thousand steps. Developers essentially never release it.

We treat it as out of scope for the main study, and record it only in Part three as a diagnostic of whether carried-over moments affect masking.

Why not just use refreshed benchmarks such as LiveBench? Fresh benchmarks help only until web crawls catch up with them, as both contamination surveys note (Xu et al.; Comprehensive Survey). They also give no statistical certificate that evaluation data are clean. Static benchmarks remain what models are compared on, so audits of them need correct calibration.

Decisive Checks

The first result that matters is the spread of the matched-inflation curves. We ask whether, at equal inflation on GSM8k items, the $\rho$ estimated from Min-K%++ and LiRA differs across optimizer settings by enough to change $m^*$ by at least a factor of two. By the scaling above, that corresponds to about a 1.4-fold difference in $\rho$.

The second result is the planner test. An optimizer-agnostic planner that falls outside the originally reported 0.785–0.92 power band on held-out optimizers, while an optimizer-aware one stays inside it, would establish the practical claim.

The third result is the reinforcement-learning factorial. There are three possible outcomes:

  • If the collapse of likelihood detectors persists with clipping off but disappears under SGD or with reset moments, masking is partly an optimizer effect. That would qualify the Fragility paper's clipping explanation.
  • If the collapse tracks only clipping, their account stands, and we have ruled out a plausible competing cause.
  • If neither factor explains it, masking is a generic effect of continued training, and both explanations need revisiting.