When a language model's training data accidentally contains test questions and their answers, its benchmark score rises without any real gain in ability. This is benchmark contamination, and it is widespread. Reported contamination rates on popular question-answering suites range from 1% to 45%, and 51 code models lost an average of 39.4% of their HumanEval score once the problems were paraphrased, per Benchmark Data Contamination of Large Language Models: A Survey. The field's response has been to detect contamination after the fact or to build fresh benchmarks. Neither response changes what a model does when it does ingest a leaked item. This proposal asks a training-side question. Can we shape a model's internal representations during ordinary training so that a leaked test item is stored by its underlying problem structure, which transfers to fresh variants, rather than as a surface-string-to-answer lookup that inflates scores?
The mechanism is contrastive learning: training the model so that the hidden representations of inputs that should be "the same" are pulled together and those that should differ are pushed apart. The positive pairs are problems with the same reasoning structure but different wording, numbers, and entity names. The hard negatives are problems with nearly identical wording but different structure. Dataset diversity enters as the source of these pairs. Procedurally generated problem families supply unlimited structure-preserving variation. Contamination enters as the stress test: we plant leaked benchmark items at a known rate and measure how much they inflate scores. The bet is that a model forced to ignore wording on clean data will gain less from memorizing leaked wording.
Core Claim
Consider an auxiliary contrastive loss over clean, procedurally generated problem families, with surface-matched hard negatives. We claim it reduces contamination inflation by more than ordinary fine-tuning on the same diverse data does. Contamination inflation is defined as the accuracy on leaked test items minus the accuracy on freshly generated items with the same structure. The contrastive loss is never applied to the leaked items, since in practice we do not know which items leaked. The effect must therefore come from a general shift in what the representation encodes.
The novelty sits exactly here. Prior work shows that diverse procedural data helps generalization, and that contaminated items leave geometric fingerprints. No prior work in this file tests whether the training objective itself changes how exploitable a leak is.
Why This Should Work, and What Constrains It
Two results supply the motivation.
Can contrastive learning avoid shortcut solutions? shows that contrastive encoders suffer from simplicity bias: if an easy feature (background color) solves the discrimination task, the encoder suppresses harder ones (shape). Its fix, Implicit Feature Modification (IFM), perturbs embeddings to weaken whatever feature currently dominates. For a memorized benchmark item, exact surface string is the easiest feature there is. Hard negatives that share the surface but differ in structure make that feature useless for the contrastive task. That is the design lever we pull, transplanted from vision to symbolic reasoning, a setting the paper explicitly leaves open.
Joint-Embedding vs Reconstruction proves, in linear models, that latent-space objectives succeed only when the augmentations vary the nuisance factor. If they do not, no amount of extra data recovers optimal downstream performance. This both motivates and constrains the design. Wording is the nuisance factor here, so our variants must vary wording aggressively while holding structure fixed. The same result also predicts that diversity along the wrong axis, such as more topics with the same templates, will not help.
LaRA finds that contaminated items produce stiff, collapsed hidden-state geometry after reinforcement-learning post-training. We read this as evidence that contamination has a representational signature that a training objective could plausibly reshape. LaRA itself only audits; it does not intervene.
DyVal generates reasoning problems from directed acyclic graphs (DAGs) and reports that fine-tuning on these structurally diverse problems improves out-of-distribution reasoning. It supplies our data generator. DyVal also defines our hardest competitor, because diversity alone might already deliver the benefit.
Research Program
We build problem families from DyVal-style DAG generators covering arithmetic, logic, and reachability. Each underlying DAG is rendered many times with different wording, variable names, numbers, and orderings. We also synthesize surface twins: renderings that keep almost all tokens of a given problem but alter one edge or operator, so the answer changes. Code families follow the same recipe, using semantics-preserving refactorings for positives and minimal semantic-breaking edits for negatives, in the spirit of the EvoEval transformations cited in the survey.
Training mixes standard next-token prediction with a sequence-level contrastive term computed on a mean-pooled hidden state $h(x)$ from a chosen middle layer:
$$ \mathcal{L} = \mathcal{L}_{\text{NTP}} + \lambda \, \mathbb{E}\left[-\log \frac{e^{s(x, x^+)/\tau}}{e^{s(x, x^+)/\tau} + \sum_{x^-} e^{s(x, x^-)/\tau}}\right] $$
The terms are defined as follows. $\mathcal{L}_{\text{NTP}}$ is the usual next-token loss on all training text. $x^+$ is a same-structure rewording of problem $x$, and the $x^-$ include its surface twins and random other problems. $s$ is cosine similarity of the pooled states, $\tau$ is a temperature, and $\lambda$ weights the auxiliary term. Because Robinson et al. report a zero-sum temperature trade-off, we also run an IFM variant of the contrastive term rather than relying on temperature tuning.
Contamination is planted deliberately. We generate a held-out test benchmark from the same generators and insert a fraction $\alpha$ of its items, with answers, into the training mix. This is label-level contamination in the survey's taxonomy, the most severe kind. The contrastive term is applied only to clean families. The design also includes a natural-benchmark arm: HumanEval items are leaked, and their EvoEval-evolved counterparts serve as the fresh variants. In both arms we sweep $\alpha$, model size (roughly 1B to 7B open models fine-tuned from a common checkpoint), and the number of exposures per leaked item.
We also add a reinforcement-learning stage. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that a brief clean RL phase drives detector AUROC from 89.1% to roughly 50–58%. We want to know whether our objective's effect survives the same treatment.
Baselines
The naive reference point is next-token fine-tuning on the original clean data with the leak planted and no diversity intervention. Its contamination inflation sets the scale of the problem. For context, the survey reports a 39.4% average drop from HumanEval to EvoEval across 51 models, and DyVal reports that phi-1.5 and WizardMath-13B score near zero on its dynamic tasks despite strong static scores. Both are observational numbers on uncontrolled models, not on our planted setup.
The strongest competitor is diversity-matched supervised fine-tuning in the style of DyVal. It trains with next-token loss on exactly the same reworded variants and surface twins, with the same token count, optimizer steps, and hyperparameter search budget, and with the same leak. It is the strongest competitor because it already exposes the model to everything the contrastive term sees. If invariance to wording is learned from data exposure alone, this baseline captures it. DyVal reports that such fine-tuning improves external reasoning benchmarks, but no reported result found on Lacuna for contamination inflation under DyVal-style fine-tuning with planted leaks. We will therefore establish this number ourselves. Beating it means reducing inflation by a statistically clear margin while matching it on fresh-item accuracy and on LiveBench. Concretely, the margin must exceed seed variance across five seeds at every $\alpha$ above 1%.
Two ablations isolate the key ingredient. The first keeps the contrastive term but replaces surface twins with random negatives. It tests whether the hard negatives that defeat the surface shortcut are what matter, as the simplicity-bias account predicts. The second keeps positives but varies only topic, not wording. It tests the Van Assel et al. prediction that diversity along the wrong axis does nothing. We also report temperature-only and IFM variants of the contrastive term.
For the detection side, we run CoDeC, which reports 99.9% dataset-level AUC, and LaRA, which reports ROC-AUC 0.79 and 31–38% true positive rate at 5% false positive rate on RL-trained models. Both are run on every trained model. Both are used as instruments, not as competitors to the training method.
Why not just…?
Why not just decontaminate the training data with n-gram filtering? Filtering only catches leaks you can string-match against a known benchmark. The survey's taxonomy makes clear that much contamination is paraphrased or semantic, and web-scale corpora contain leaks nobody has catalogued. Our approach is meant to be complementary: insurance for the leaks that filtering misses. We will include an arm in which leaked items are lightly reworded before insertion, so that n-gram filtering demonstrably fails, and check whether the contrastive model's advantage holds there.
Why not just train on diverse variants with ordinary fine-tuning, as DyVal does? This is the real threat. It may simply work as well. The contrastive term's specific claim is that surface twins teach the model that wording is not predictive, whereas next-token training on twins teaches only that different strings have different answers. The diversity-matched baseline above settles the question directly. If it closes the inflation gap as well as our method does, the conclusion is that diversity of the right kind is sufficient, and the contrastive machinery is unnecessary. That would still be a useful, publishable result about contamination robustness. We will report it as such.
Why not just evaluate on dynamic benchmarks like LiveBench or DyVal and ignore contamination? Dynamic benchmarks fix measurement, not models. Static suites remain the default for reporting. Contamination also matters downstream: a model that answered a leaked problem by recall will fail on the deployed variant, as MLAgentBench and Unveiling Causal Reasoning in Large Language Models show for post-cutoff tasks. We use LiveBench to confirm that the contrastive term does not harm general capability.
Why not just rely on a detector such as CoDeC or LaRA? The fragility paper shows that likelihood-based detection collapses to near chance after clean RL. LaRA's best combined score, ROC-AUC 0.79, leaves most leaks undetected at practical false-positive rates. A detector also flags a problem without removing it.
There is a real interaction to watch. If our objective removes the geometric stiffness LaRA relies on while leaving inflation intact, we would have built a cloaking method. That is why inflation is the primary metric and detection AUC is reported alongside it.
Proposed Experiments and Decisive Checks
The primary outcome is contamination inflation versus leak fraction $\alpha$, plotted for each training condition, with fresh-item accuracy and LiveBench scores as guards against capability loss.
When Is Benchmark Contamination Detectable? shows that audit power scales as $\alpha \rho \sqrt{m}$, where $\rho$ is behavioral separability and $m$ is the number of evaluated items. We therefore choose $m$ so that the naive baseline's inflation is detectable at 80% power. This ensures that a small measured inflation for our method is a real reduction rather than an underpowered null.
The decisive check has two parts. First, at fixed data and compute, does the hard-negative contrastive model show lower inflation than diversity-matched fine-tuning? Second, does that gap disappear when surface twins are replaced by random negatives? A yes on the first and a disappearance on the second would confirm the mechanism.
Secondary checks probe the representations directly. We test whether leaked items' pooled hidden states cluster with their structure-mates or with their verbatim copies, and whether LaRA's stability index and CoDeC's confidence shift move with inflation or decouple from it. We then repeat everything after a clean RL phase to see whether the reduction persists under the conditions that defeat current detectors. Finally, the HumanEval/EvoEval arm tests whether gains on synthetic DAG families carry over to a natural, widely used benchmark.
References
- Benchmark Data Contamination of Large Language Models: A Survey
- Can contrastive learning avoid shortcut solutions?
- Joint‑Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self‑Supervised Learning
- LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Training
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Detecting Data Contamination in LLMs via In-Context Learning
- MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?