When Hyperparameter Search Tunes for Memorization

Automated hyperparameter search picks a training configuration (learning rate, adapter size, number of epochs) by trying many configurations and keeping the one that scores best on a validation set. For language models, that validation set is often a public benchmark such as MATH-500 or AIME, and the base model may already have seen those questions and answers during pre-training. This proposal asks a simple question. When the validation signal is partly memorized, does the search reliably drift toward configurations that preserve or amplify recall instead of configurations that teach the skill? If it does, how much generalization does that cost, and which search methods are hurt most?

We make one testable claim. Contamination is not random noise on the validation score. It is a structured bonus that depends on the hyperparameters, so a better optimizer will find and exploit it more efficiently. This predicts the opposite of the reassuring finding that Bayesian optimization overfits validation noise less than random search does. We test the claim with contamination we inject ourselves, so the ground truth is known. We then check whether a cheap fix works: scoring each trial on freshly generated, solver-verified problem instances. The fix should be compared against the more obvious remedies of filtering suspicious items or holding out a fixed clean set.

Core Claim and Why It Is Open

Write the validation score of configuration $\lambda$ as $V(\lambda) = S(\lambda) + M(\lambda) + \varepsilon(\lambda)$. Here $S$ is genuine task skill, $M$ is the accuracy gained by recalling memorized benchmark items, and $\varepsilon$ is mean-zero sampling noise. Overtuning in Hyperparameter Optimization studies only the $\varepsilon$ term. It shows across tabular and deep-learning HPO suites that optimizing a noisy validation score can pick worse configurations. It reports that about 10% of runs are severe, and that in about 10% of cases the chosen configuration generalizes worse than the default. That paper explicitly assumes clean, disjoint validation and test splits. Its finding that surrogate-based optimizers act as smoothers applies to $\varepsilon$, and there is no reason it should apply to $M$.

Our hypothesis is that $M$ varies systematically with hyperparameters. For example, gentle updates and few epochs leave pre-trained recall intact, and large adapter ranks with little regularization strengthen it. Under that hypothesis, an optimizer that models $V$ well will climb toward high-$M$ regions.

The contamination literature supplies the evidence that $M$ is large but never varies the tuning procedure:

  • Reasoning or Memorization? shows that Qwen2.5 reconstructs 65.8% of MATH-500 answers exactly from an 80% question prefix. It also shows that reinforcement learning with random or inverted rewards still improves contaminated scores. It treats all training hyperparameters as fixed controls.
  • On The Fragility of Benchmark Contamination Detection in Reasoning Models injects contamination deliberately. Fine-tuning-stage contamination inflates pass@1 by 8.8% on average across math benchmarks. A single hyperparameter, the policy-ratio clipping threshold in GRPO-style reinforcement learning, pushes loss-based detectors down to roughly 50–58% AUROC. It evaluates fixed configurations rather than a search.

On the tuning side, two systems optimize directly on benchmarks of this kind without any contamination audit. MAGO replaces a hand-set reward weight with Pareto search and reports accuracy on MATH-500 and AIME 2024. AutoTuneX runs bandit-style search around default fine-tuning settings.

The research file contains no paper that measures what a search procedure selects when its validation signal carries $M$. That measurement is the contribution.

Research Program

Contamination with known ground truth. We use a base model with a documented training cutoff, run in two conditions:

  • Natural contamination: Qwen2.5-Math at the 1.5B and 7B scales, whose MATH-500 leakage is documented.
  • Controlled contamination: a Llama-family model into which we inject a random half of MATH-500 and AIME items (question plus solution) by supervised fine-tuning. We follow the Stage I protocol of the Fragility paper, which gives us a known contaminated and uncontaminated half of each benchmark.

Tuning task. The downstream task is LoRA fine-tuning (low-rank adapter training) followed optionally by short GRPO reinforcement learning on a clean math training set. The search space covers:

  • learning rate,
  • LoRA rank,
  • epochs,
  • weight decay,
  • the GRPO clipping threshold,
  • sampling temperature.

Measuring selection harm. Each search method runs under a fixed trial budget, and the selected configuration is scored on held-out targets that no model can have memorized. We define contamination-induced regret as

$$R = T(\lambda^*_{\text{clean}}) - T(\lambda^*_{\text{contam}}),$$

where $T$ is accuracy on the uncontaminated test targets. $\lambda^*_{\text{contam}}$ is the configuration selected using the contaminated validation half. $\lambda^*_{\text{clean}}$ is the one selected using an equally sized uncontaminated half.

Mapping the memorization bonus. We estimate $M(\lambda)$ directly by scoring each configuration on the contaminated half minus the uncontaminated half. This shows which hyperparameters drive recall, and whether search trajectories move toward high-$M$ regions over trials.

Search procedures compared.

Proposed fix. Every trial is validated on a new draw from a procedural generator: BeyondBench's solver-verified tasks, DyVal's graph-generated reasoning problems via PromptBench, or RandomCalculation. Fresh draws make $M = 0$ by construction. Because the optimizer never sees the same items twice, it also cannot overfit $\varepsilon$ to a fixed set, which Overtuning identifies as the danger with small validation sets of 500 items or fewer.

Main open risk. Procedural tasks may measure a different skill from competition math. We therefore report the rank correlation between generator scores and post-cutoff math scores across configurations. A low correlation would mean the fix removes contamination but tunes for the wrong target.

Baselines

Default configuration. The naive reference point is the untuned default configuration of the fine-tuning recipe. Its value is that Overtuning found about 10% of searches end up worse than the default. Any search arm that falls below the default on the uncontaminated targets is doing active harm.

Status quo. The strongest existing practice, and the one the proposal must beat, is TPE or Gaussian-process Bayesian optimization validated on the public benchmark as-is. This is what MAGO- and AutoTuneX-style pipelines do in effect. It is the strongest competitor because model-based search is the method shown to beat expert hand tuning (Practical Bayesian Optimization) and to resist validation noise better than random search (Overtuning). No reported result on Lacuna quantifies HPO regret under contaminated validation. The closest numbers are Overtuning's roughly 60/30/10 split of no/mild/severe overtuning on clean tabular and deep-learning suites.

Fairness conditions. All arms share the same base model, training data, search space, trial count and GPU-hour budget. Validation sets are size-matched, and results are averaged over at least five search seeds.

What counts as beating the status quo. Procedural fresh-draw validation beats it if two things hold. First, its selected configurations score higher on post-cutoff targets by more than the seed-to-seed spread. Second, its $R$ is not significantly different from zero, while the status-quo arm's $R$ is.

Detection-filtered validation. This arm drops validation items flagged by partial-prompt completion or Min-K% scoring before the search begins.

Fixed clean holdout. This arm validates on a fixed, size-matched set of clean items.

Freshness ablation. This arm uses a single fixed draw from the same procedural generator, reused across all trials. It isolates the key ingredient. If fixed-draw and fresh-draw arms tie, the benefit comes from removing $M$ alone, not from resampling.

Upper bound. Selection by test score gives an oracle ceiling.

Why not just…?

Why not just validate on a clean holdout, such as LiveBench's post-cutoff items? This is the real threat, and we treat it as a first-class arm. LiveBench refreshes monthly, but any fixed month is a small, reused set. That is exactly the regime where Overtuning finds noise overfitting. Its freshness also decays once the set is published. If the fixed clean holdout matches fresh procedural draws on post-cutoff accuracy, we will report that the lesson is simply "use any clean set." In that case the contribution narrows to measuring how large $R$ is for the status quo.

Why not just filter contaminated items with a detector first? For contamination introduced only by fine-tuning, detectors may work well enough, and the detection-filtered arm will show this. However, the Fragility paper reports detector AUROC collapsing to about 50–58% once clean reinforcement learning follows contamination. Our search space includes the clipping threshold that causes that collapse, so the filter's reliability itself depends on $\lambda$. We report filtered-arm regret separately for configurations with and without the reinforcement learning stage.

Why not rely on Bayesian optimization's smoothing, or on cross-validation? Smoothing averages out mean-zero noise. Our claim is that $M$ is a consistent signal a surrogate will model and pursue. The direct test is whether $R$ is larger for TPE and Gaussian-process search than for random search at equal budget, which would reverse the ordering Overtuning found for noise. Resampling the validation benchmark into folds is cheap because it needs no retraining, and we include it. But under natural contamination every fold is memorized, so we expect it to leave $R$ unchanged.

Decisive Checks

The central result has three parts:

  • the sign and size of $R$ for each search method under controlled contamination;
  • whether search trajectories measurably climb $M(\lambda)$ over trials;
  • whether model-based search shows larger $R$ than random search.

Evaluation targets are RandomCalculation, the BeyondBench Easy and Hard suites, DyVal levels D1 to D4, and post-cutoff AIME 2025 and LiveBench math.

The hypothesis is falsified if two findings hold together. First, $R$ is indistinguishable from zero across methods. Second, $M(\lambda)$ is flat across the search space, meaning contamination inflates every configuration equally and leaves the selection unchanged. A secondary check reruns the MAGO-style reward-weight search with fresh-draw validation. It tests whether the Pareto point chosen on MATH-500 stays the same when memorization is removed. MAGO reports 2.2–3× token reduction and up to 9.4% higher accuracy on the static benchmarks.