A common argument for letting language models check and repair their own code is that checking is easier than writing. On hard LiveCodeBench problems, one study reports that models write correct code only 32.4% of the time but judge candidate code correctly 76.5% of the time (PGS). That gap between writing and checking is called the verification-generation asymmetry, and it motivates a family of systems in which a model writes code, writes tests or properties for it, runs them, and edits the code over several turns (ReVeal). These loops are evaluated on public contest problems. For those problems, the tests, the input-output examples, and the accepted solutions are all scraped into pretraining data. No one has asked whether the checking advantage is itself a product of that leakage. A model that has seen a problem's test file can "verify" by recognition rather than reasoning. The same model can then use its own error messages as cues for recalling a solution it saw during training.
This note proposes to measure the asymmetry and the gains from iterative repair while controlling which artifacts leaked. The core design fine-tunes open models on selected problems. For some problems we inject only the solutions, for some only the tests, and for some the paired problem and solution. The rest stay clean. We then compare writing accuracy, checking accuracy, and repair gains within the same model. The design is paired with a cheap probe that works on closed models: before the checker sees the problem, the task statement is rewritten into an unfamiliar story. The deliverable is a contamination-corrected estimate of how much of the checking advantage, and how much of the benefit of feedback loops, survives on problems the model cannot have memorized.
Core Claim
We claim that current estimates of the verification-generation asymmetry in code are confounded by contamination, and that the confound acts through different channels for checking and for writing. The prediction is directional and testable. Leaked tests should inflate checking accuracy while leaving writing accuracy nearly untouched. Leaked paired solutions should inflate writing and should also make execution feedback in repair loops act as a retrieval cue, not a debugging signal. If the claim is right, part of the reported benefit of verifier-guided refinement is recall that only looks like reasoning. If it is wrong, and the asymmetry and repair gains are flat across contamination conditions, then we have the first evidence that these methods are robust to contamination, which the field currently assumes without testing.
The quantity we track for each model and condition $c$ is the asymmetry gap
$$\Delta_c = V_c - G_c,$$
where $G_c$ is single-attempt pass rate on problems in condition $c$ and $V_c$ is balanced accuracy at labeling candidate programs as correct or incorrect. We also track the repair gain $R_c$, defined as final minus first-turn pass rate after a fixed number of feedback turns. The central test is whether $\Delta_c$ and $R_c$ differ between leaked and clean problems for the same model.
What Exists and What Is Missing
Two systems motivate this work by assuming the asymmetry is real and building on it. PGS has a separate tester model extract invariant properties from the prompt and feed back only the smallest failing input. It reports bug-fix rates up to 67.9% on HumanEval. ReVeal trains a single agent with reinforcement learning to alternate between writing code and writing tests, and reports gains that hold over 25+ turns. Both evaluate on HumanEval, MBPP, CodeContests and LiveCodeBench without separating memorized from reasoned verification. We reuse their loops as the objects of measurement; we do not propose a competing loop.
A second line constrains the design by showing that the tests themselves are suspect. Rethinking Verification for LLM Code Generation shows that model-written tests tend to catch only the kinds of bugs models make, and builds TCGBench with an average of 36 incorrect human submissions per problem. We use that pool as a realistic set of wrong candidates for measuring $V_c$, since model-generated wrong answers would share the checker's blind spots.
Contamination work supplies the tools. LiveCodeBench timestamps problems so that models can be scored only after their training cutoff, and includes a single-turn self-repair task. DyCodeEval rewrites a problem's story while keeping its algorithm. It shows that Llama-3.2-1B fine-tuned on leaked benchmarks jumps from about 20% to about 80% on static problems but stays near 13% on rewritten variants. Its fine-tuning protocol is the template for our injection experiments. Quantifying Contamination matches benchmark solutions against StarCoderData using syntax-tree similarity. It finds exact matches for 20.8% of MBPP and a 72% versus 22% accuracy split between most- and least-similar problems for StarCoderBase-15.5B. We use its matching method to measure how closely repair trajectories converge to memorized code.
Rethinking the effects of data contamination in Code Intelligence separates input-only, output-only and paired leakage, but only with text-overlap metrics. It finds that only paired leakage inflates scores and that fine-tuning acts as a "reset button." That result appears to contradict DyCodeEval and the overfitted models in LiveCodeBench. Our injection design addresses the contradiction directly: we inject different artifact types, measure with execution-based correctness, and add downstream fine-tuning afterwards to test whether the "reset" holds for functional correctness and for checking.
No paper in this set evaluates the asymmetry across a temporal cutoff, across story rewrites, or under controlled leakage. None asks whether multi-turn feedback lets a model get past a story rewrite to a memorized solution.
Research Program
The program has three arms that cross-validate one another.
The causal arm uses open models at 1B–7B scale. Starting from a pool of HumanEval, MBPP-Sanitized and post-cutoff LiveCodeBench problems, we randomly assign each problem to one of four conditions: clean, tests-only leaked, solution-only leaked, or problem-plus-solution leaked. Leaked artifacts are mixed into a continued-training corpus at a fixed repetition rate. Random assignment means difficulty is balanced across conditions by construction. This is what temporal splits cannot provide, because newer contest problems may simply be harder.
We then measure three quantities per condition:
- $G_c$, the single-attempt pass rate.
- $V_c$, checking accuracy, measured two ways: as a direct correct/incorrect judgment on a balanced set of correct programs and TCGBench-style incorrect submissions, and as the kill rate of the tests or properties the model writes.
- $R_c$, the repair gain under three loops: raw-traceback repair, a PGS-style property-and-minimal-counterexample loop, and best-of-N with self-written tests.
A second round applies ordinary instruction fine-tuning on unrelated code data after injection, to test the "reset button" claim.
The probe arm has to work on models whose training data we cannot control. The checker receives a DyCodeEval-style rewrite of the problem statement, while the code under review is left unchanged (or has its variable names mechanically renamed to match). If checking accuracy drops sharply under the rewrite while writing accuracy on the same rewrite drops less, the checker was relying on surface recall. The causal arm calibrates this probe: on open models we know the ground-truth leakage, so we can report how well the rewrite-induced drop in $V$ predicts assignment to a tests-leaked condition.
The trajectory arm addresses whether feedback helps a model get around a story rewrite and reach memorized code. For solution-leaked problems presented as rewrites, we record every intermediate program in a repair loop. We compute its syntax-tree similarity to the injected solution, using the Dolos-based matching from Quantifying Contamination, and compare it against the similarity for clean problems solved to the same final correctness. The signature of recall is a sudden jump toward the injected solution right after an error message, rather than gradual local edits.
Baselines
The do-nothing reference is the uncorrected asymmetry: generation and verification accuracy measured on the full public benchmark, as current work reports it. For LiveCodeBench Hard, the reported figures are 32.4% generation and 76.5% verification from PGS. We re-measure this on our models with the same prompts. Our corrected estimate "beats" it if the clean-condition gap $\Delta_{\text{clean}}$ differs from the pooled gap by more than a bootstrap 95% interval.
The naive refinement baseline is raw-traceback self-repair as in LiveCodeBench, which reports GPT-4-Turbo improving from 24.5% to 36.9%. It isolates whether any contamination effect on repair gain is specific to verifier-guided loops or appears with plain error messages. It runs with the same turn budget and the same models as the other loops.
The strongest existing refinement method we can run fairly is a PGS-style property tester with minimal counterexamples. We also run a ReVeal-style interleaved write-and-test loop, using released weights if available and a prompted approximation otherwise. The reported reference figure is ReVeal's Pass@1 on LiveCodeBench V6 rising from 34.8% at turn 1 to 38.7% at turn 25 with a Qwen2.5-32B base, and 33.6% on CodeContests against 13.3% for the base instruct model (ReVeal). All loops get identical per-problem token budgets and turn caps (3 and 10 turns) and a shared prompt-tuning budget, tuned on a held-out clean split only.
The key ablation is the artifact type of the injection. Comparing tests-only against solution-only leakage is what separates "checking is inflated by recalled tests" from "everything is inflated by recalled solutions."
Why not just…?
Why not just rerun PGS and ReVeal on post-cutoff LiveCodeBench problems? This is the cheapest serious alternative, and we include it as an observational check. It cannot identify the mechanism. A drop after the cutoff mixes contamination with differences in problem difficulty and platform mix. It also cannot say whether the leaked artifact was the tests or the solution. Random assignment in the causal arm removes the difficulty confound. If the temporal split and the injection experiments agree, the cheap method is validated for future use.
Why not just use DyCodeEval rewrites on their own? Rewriting shows that single-attempt generation relies on the story, but it has never been applied to checking or to multi-turn loops. On its own it cannot tell whether a drop reflects lost recall or a rewrite that happens to be harder. We use it as the probe and calibrate it against known leakage. If the probe alone recovers the causal-arm effects, that is a positive result: it means closed models can be audited without retraining.
Why not just split problems by similarity to the pretraining corpus, as Riddell et al. did? That approach needs access to the pretraining corpus, which rules out most frontier models. Similarity also correlates with problem popularity and simplicity: the 72% versus 22% gap in Quantifying Contamination is partly a difficulty gap. We use it as a third observational split on StarCoder-family models. We treat agreement with the injection results as corroboration, not as a substitute.
Why not just use best-of-N sampling and skip the loops? This alternative threatens the value of refinement, not the measurement itself. Best-of-N with self-written tests is one of our three loops. If its contamination sensitivity matches that of the multi-turn loops, the conclusion is that the self-written tests are the vulnerable component, whatever the search strategy.
Proposed Experiments and Decisive Checks
The first decisive check is H1. In the causal arm, tests-only leakage should raise $V_c$ by a margin that exceeds its effect on $G_c$, so that $\Delta_{\text{tests}} > \Delta_{\text{clean}}$ with non-overlapping bootstrap intervals, pooled over at least three model sizes. A null here, with the gap flat across conditions, is the result that would license the verification-generation asymmetry as a contamination-robust basis for the design of feedback loops.
The second check is H2. For solution-leaked problems presented as rewrites, repair gain $R_c$ should exceed single-attempt gain by more than it does on clean rewrites, and trajectories should show the sudden jump in similarity to the injected code. This would show that feedback lets a model get past story rewrites to memorized solutions. It would also mean that rewrite-based benchmarks like DyCodeEval need to be re-run with repair loops before being trusted as contamination-free.
The third check addresses the fine-tuning tension. After post-injection instruction tuning, either the inflation persists, which reconciles the literature in DyCodeEval's favor for execution-based metrics, or it disappears, which supports Yang et al..
Finally, we report the calibration of the rewrite probe: its area under the curve for detecting tests-leaked problems in the causal arm. We then apply it to closed models on HumanEval, MBPP and LiveCodeBench, producing contamination-adjusted asymmetry estimates for systems that cannot be retrained. No reported result was found on Lacuna for verification accuracy under controlled leakage or under story rewrites, so these experiments start from zero prior numbers on the central quantities.
References
- PGS: Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback
- ReVeal: Self-Evolving Code Agents via Reliable Self-Verification
- Rethinking Verification for LLM Code Generation: From Generation to Testing
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
- Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models
- Rethinking the effects of data contamination in Code Intelligence