Does Memorization Masquerade as Self-Repair? Contamination in Sparse and Dense Code Models

Code models are increasingly judged by how well they fix their own programs. A model writes a program, runs it against tests, reads the error message, and tries again. The headline numbers for this ability, however, come mostly from benchmarks such as HumanEval and MBPP, and large fractions of those benchmarks sit verbatim in public training corpora. One audit found that 20.8% of MBPP problems appear exactly in StarCoderData (Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models). Nobody has checked whether an error message helps a model reason toward a fix, or whether it simply prompts the model to recall an answer it already memorized. The question gets sharper for mixture-of-experts (MoE) models, which route each token through a few of many sub-networks ("experts"). Their total parameter count, and with it their capacity to store memorized text, can grow without any increase in the compute spent per token.

We propose a controlled experiment to answer this. We pretrain matched dense and MoE code models while planting a randomly chosen half of each benchmark into the training data. We then measure how much of each model's self-repair gain on planted problems exceeds its gain on unplanted problems of identical difficulty. Two results would follow. The first is a causal estimate of how contamination inflates iterative-refinement scores, broken down by architecture. The second is a test of whether the shape of a repair trajectory can serve as a contamination detector for models whose training data cannot be inspected. A memorized fix tends to jump abruptly to a near-canonical solution, while a reasoned fix makes small, local edits.

Core Claim

Two findings in the research file point toward the claim. Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks reports that factual recall scales with total parameters, while multi-step reasoning tracks active parameters per token. Separately, memorization of leaked benchmarks is known to grow with model size (Quantifying Contamination…, The Impact of Post-training on Data Contamination).

Putting these together gives our hypothesis. At a fixed number of active parameters, a sparser MoE should memorize more of a leaked benchmark. Its measured self-repair gain on that benchmark should therefore be more inflated than a dense model's. On clean problems, the opposite may hold: data-starved experts may repair worse than the dense model. If this is true, current static-benchmark comparisons of sparse and dense coders would systematically overstate the refinement ability of sparse models.

We define the repair inflation after $r$ feedback rounds as

$$\Delta_r = \big(P_r^{c} - P_0^{c}\big) - \big(P_r^{u} - P_0^{u}\big),$$

where $P_r^{c}$ and $P_r^{u}$ are pass rates after $r$ rounds on contaminated and uncontaminated problems, and round 0 is one-shot generation. Because the planted half is chosen at random, $\Delta_r$ estimates the causal effect of contamination on what repair adds, beyond the one-shot gap that prior work already measured.

Research Program

The pretraining setup extends the controlled-injection design of The Impact of Post-training on Data Contamination. That paper planted 5 copies of GSM8K and MBPP in a 25B-token stream of dense models up to 4B parameters and studied only one-shot generation.

We add three things. The first is an architecture axis: dense models paired with MoE models at matched active parameters, sweeping expert count and the number of experts used per token, following the design space of the Optimal Sparsity paper. The second is a contamination dose of 0, 1, or 5 copies. The third is a planted benchmark that is itself free of web contamination: half of a post-cutoff slice of LiveCodeBench together with half of MBPP. The LiveCodeBench half matters because MBPP is already leaked on the web, which would muddy the "uncontaminated" arm. For LiveCodeBench, we control both arms completely.

After pretraining, every model receives identical clean post-training. One arm uses supervised fine-tuning on execution-feedback dialogues, the Code-Feedback data from OpenCodeInterpreter, filtered against both benchmarks. The other arm uses multi-turn reinforcement learning (GRPO with test-pass rewards). We include both because the post-training paper found that supervised fine-tuning extracts memorized answers while GRPO generalizes them. We want to know whether that split holds once feedback loops are added.

We evaluate three refinement protocols. The first is LiveCodeBench-style single-round self-repair. The second is the S* pipeline, which samples many candidates in parallel and then runs two debugging rounds against public tests. The third extends this to five rounds, so we can see whether the 2–3-round plateau reported by Yan et al. differs between planted and clean items.

On every trajectory, we record three trajectory features. The first is the edit distance between successive attempts. The second is the abstract-syntax-tree similarity of each attempt to the canonical solution, computed with the Dolos-style matcher from the contamination audit. The third is the per-round log-likelihood the model assigns to the gold solution, the predictor shown to be stable by Pretraining Scaling Laws for Generative Evaluations of Language Models. We fit a small classifier on these features to predict planted versus clean items. We then transfer it unchanged to public open-weight models and validate it against LiveCodeBench release dates. A detector that works should fire on pre-cutoff problems and fall silent after the cutoff, in line with the "contamination cliff."

Baselines

The naive reference point is to report self-repair gain on the full benchmark, ignoring contamination. This is how OpenCodeInterpreter reports its HumanEval numbers: 79.3 single-turn, 82.9 with execution feedback, and 88.4 with simulated human feedback. If $\Delta_r$ comes out near zero, this naive practice is vindicated, and our main hypothesis fails.

The strongest existing way to measure refinement without contamination is to evaluate only on post-cutoff LiveCodeBench problems. That benchmark reports GPT-4-Turbo self-repair rising from 24.5% to 36.9% Pass@1, and S* reports R1-Distill-Qwen-32B rising from 69.1% to 85.7%. We run this baseline on the same checkpoints and the same clean problems. Our study beats it only in explanatory power, by showing which architectures would have been misranked on static benchmarks. It is a reference measurement, not a competitor to outscore.

For detection, the strongest competitor is the dual Levenshtein/AST corpus-overlap method from Quantifying Contamination…. It separated StarCoderBase-15.5B accuracy into 72% on the most-overlapping MBPP decile and 22% on the least-overlapping decile. That method needs corpus access, and in our controlled setting it is effectively an oracle upper bound. The fair black-box competitor is a one-shot likelihood probe: the gold-solution log-likelihood alone, with no feedback rounds. Both detectors get the same held-out problems and the same calibration split. No reported result was found on Lacuna for a detection AUC of either method on code benchmarks. We therefore measure both ourselves. The trajectory detector counts as winning if it beats the one-shot likelihood probe in AUC on planted items at 1-copy dose, where memorization is faint, and still recovers the temporal cliff on public models.

The ablation that isolates our key ingredient feeds the trajectory classifier only round-0 features. If round-0 features suffice, the feedback loop adds no detection signal.

Why not just…?

Why not just evaluate on fresh LiveCodeBench problems and ignore contamination? For clean leaderboard numbers, this is the right answer, and we use it. It cannot tell whether the field's existing static-benchmark conclusions about refinement, for example in OpenCodeInterpreter, were inflated, or by how much per architecture. It also cannot audit models already deployed against older benchmarks. Fresh benchmarks also carry real maintenance costs, as the LiveCodeBench authors note. Our randomized planting answers the causal question that fresh evaluation sidesteps.

Why not just run corpus-overlap detection? It works when the corpus is available, and in our setting it serves as the oracle. For models whose training data cannot be inspected, it is impossible. The trajectory detector exists only for that case. We grade it against corpus overlap in the controlled models before trusting it on public ones.

Why not just use one-shot likelihood or perplexity probes? This is the real threat. Gold-solution likelihood is already known to track ability closely, so it may track memorization just as well. The round-0 ablation settles it directly. If one-shot likelihood matches the trajectory detector at every dose, we drop the detector claim and keep the architecture-by-contamination scaling result as the contribution.

Why not just study repair scaling across sparsity on clean data? That answers half of Gap 1 and is contained in our clean arm. It cannot explain why sparse models might look better on static suites than on fresh ones. That discrepancy is exactly what the planted arm measures.

Decisive Checks

The main result is the sign and size of $\Delta_r$ as a function of expert count at fixed active parameters, at 1 and 5 copies, under both post-training arms. Our hypothesis predicts that $\Delta_r$ grows with total parameters, and grows more under supervised fine-tuning than under GRPO. It also predicts that clean-arm repair gain tracks active rather than total parameters, consistent with the finding in Optimal Sparsity that inference-time compute cannot rescue a poorly chosen sparsity level.

The hypothesis is falsified if $\Delta_r$ is near zero for all architectures. That outcome would mean feedback loops do not amplify contamination beyond its one-shot effect. It would be a useful result in its own right, because it would legitimize static-benchmark refinement studies.

A second check concerns trajectory shape. Planted items should show a large single-round jump in similarity to the canonical solution, while clean items should show incremental edits.

The final check is external. The detector, fit only on our controlled models, must reproduce the release-date cliff on public open models without retraining.