Code models are increasingly deployed with an abstain option. The model emits a program only when its confidence clears a threshold, and that threshold is tuned on a labeled benchmark so that, say, at most 10% of emitted programs fail their tests. The tuning almost always uses HumanEval, MBPP or APPS, and these are exactly the benchmarks known to have leaked into pretraining corpora. A model that has memorized a problem is confident and correct for a reason that does not carry over to new problems. A threshold fitted on such data therefore learns that high confidence means correct. On genuinely new code, that rule may be false.
This note proposes two things. First, an audit: does the error-rate guarantee of confidence-based abstention for code survive the move from contaminated benchmarks to problems released after a model's training cutoff? Second, a fix: compute confidence on an automatically rewritten version of each problem, both when fitting the threshold and when deploying it, so the score reflects solving rather than recall. Old, leaked benchmarks then become usable calibration data again, without a constant supply of freshly labeled problems.
Core Claim
We claim that contamination inflates confidence and correctness together on the same items. This makes thresholds and recalibration maps fitted on contaminated benchmarks optimistic in a predictable direction. The bias is largest in the high-confidence region, which is where abstention decisions are made.
The prediction is concrete. Consider a threshold calibrated on HumanEval or MBPP to keep the false discovery rate (FDR, the fraction of emitted programs that fail their tests) below a target α. That threshold should exceed α on LiveCodeBench problems released after the model's cutoff, and the excess should grow with the model's measured contamination.
We further claim that a variant-anchored confidence removes most of this gap. This is the model's confidence on a meaning-preserving rewrite of the problem, with renamed identifiers and a new surface narrative but the same algorithm and the same tests.
Write $x$ for a problem, $\tilde{x} = R(x)$ for its rewrite by a fixed rewriter $R$, $y$ for the generated program and $c(\cdot)$ for a confidence score such as the sum of token log-probabilities. The emission rule is to output $\tilde{y} \sim M(\tilde{x})$ only if $c(\tilde{x}, \tilde{y}) \ge \tau$. The threshold $\tau$ is chosen on rewritten calibration problems so that
$$\Pr\big[\tilde{y} \text{ fails tests} \mid c(\tilde{x},\tilde{y}) \ge \tau\big] \le \alpha .$$
The guarantee is only as valid as the assumption that the calibration problems and the deployment problems look alike to the model. The proposal's bet is that rewriting makes old and new problems look alike, because it strips away the cue that memorization relies on. This is cheaper than the alternative of sourcing new problems.
Prior Work and Where This Sits
Two papers supply the starting point, and both leave contamination untested. Towards Functional Correctness of Large Code Models with Selective Generation builds selective code generation. It strengthens each problem's tests by fuzzing, generating 600+ input-output pairs from the reference solution, and it calibrates an abstention threshold with a statistical guarantee that FDR stays below a user target. However, its calibration and test splits both come from HumanEval-f, MBPP-f, APPS-f and Mercury-f, so the guarantee is checked only inside a possibly contaminated distribution. We reuse its fuzzed test construction and its threshold procedure, and we change the distribution on which the guarantee is checked.
Calibration and Correctness of Language Models for Code measures how well token-probability and verbalized confidence track test outcomes on HumanEval, MBPP, DyPyBench and Defects4J. It reports out-of-the-box expected calibration error (ECE, the average gap between stated confidence and actual accuracy) of 0.09 to 0.73. Platt scaling, a one-variable logistic fit, cuts average ECE from 0.32 to 0.03. The same paper warns that Platt scaling often collapses confidences toward the base rate, which keeps ECE low while destroying the ability to separate successes from failures. That warning constrains our design: we report ranking quality (AUROC) and emitted coverage alongside ECE, never ECE alone.
Three papers motivate the claim by showing that contamination inflates correctness on code. Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models measures overlap between benchmarks and pretraining corpora using edit distance and syntax-tree matching. It finds 20.8% of MBPP verbatim in StarCoderData, and StarCoderBase-15.5B reaches 72% accuracy on the most-overlapping tenth of MBPP versus 22% on the least. LiveCodeBench timestamps contest problems and shows pass rates dropping sharply after a model's cutoff. DyCodeEval rewrites a problem's story while keeping its algorithm. It shows a Llama-3.2-1B fine-tuned on leaked HumanEval rising from roughly 20% to 80% on the originals while scoring about 13% on rewrites. None of these three records confidence. We use LiveCodeBench as the held-out truth and DyCodeEval's rewriting idea as the core of our method.
Two papers suggest that confidence itself carries a contamination signal. Data Contamination Calibration for Black-box LLMs detects training-set membership from how sharply confidence drops when a text is perturbed. Detecting Functional Memorization in Code Language Models shows that code can be memorized at the level of logic even when the surface text differs. This matters for us as a threat: if memorization survives rewriting, rewriting cannot fully remove it. We test this directly.
Finally, SALT uses procedurally generated, contamination-proof tasks and finds that reasoning models fail with high confidence. It also finds that summed log-probabilities rank errors best, which is why we adopt that score as default. SALT studies predicting the output of given code rather than writing programs, and it never compares contaminated with clean items. Our study does both.
Research Program
The work has three stages.
Stage one: measuring the gap. For each open-weights model, we generate programs for:
- HumanEval and MBPP-Sanitized, with fuzzed test suites built as in the selective-generation paper;
- LiveCodeBench problems, split at the model's reported cutoff.
We record summed and average token log-probabilities and verbalized confidence. We then fit both Platt maps and FDR thresholds on the old benchmarks and evaluate them on post-cutoff LiveCodeBench, measuring realized FDR, coverage, AUROC and ECE. Contamination per item is estimated from corpus overlap where the corpus is public (StarCoderData for StarCoderBase) and from confidence-under-perturbation scores otherwise.
Stage two: a controlled causal check. Following DyCodeEval's leak experiment, we fine-tune a small model on a random half of HumanEval and MBPP. This gives a known, item-level contaminated set within the same distribution, against an uncontaminated half. Stage two separates the effect of leakage from differences in difficulty between old benchmarks and new contest problems. That confound would otherwise undermine stage one, since LiveCodeBench problems are harder.
Stage three: variant-anchored calibration. To keep cost low and reproducible, we use a fixed open-weights rewriter rather than a commercial four-agent pipeline. DyCodeEval reports that LLM rewriting has real API cost and can drift in meaning. Rewrites are checked by running the reference solution against the original tests, and we hand-audit a sample against DyCodeEval's 95% human-verified consistency figure.
We compare three placements of the rewrite:
- rewriting only the calibration set;
- rewriting both the calibration set and the deployment queries;
- an ensemble that averages confidence over several rewrites.
Baselines
The naive reference point is the model's raw confidence with a fixed threshold and no calibration. Its expected failure mode is the out-of-the-box ECE range of 0.09 to 0.73 reported by Calibration and Correctness of Language Models for Code.
The first strong baseline is Platt scaling fitted on HumanEval/MBPP, the standard post-hoc fix. Its reported result is average ECE reduced from 0.32 to 0.03 on in-distribution benchmarks, from the same paper. We fit it on exactly the calibration problems our method uses and evaluate on the same post-cutoff LiveCodeBench problems. Beating it means lower post-cutoff ECE at equal or better AUROC, so that a win cannot come from collapse to the base rate.
The strongest existing method for the abstention question is selective code generation from Towards Functional Correctness of Large Code Models with Selective Generation. The file reports that it keeps FDR below the target, for example under a 30% limit, on all its fuzzed benchmarks. No reported result was found on Lacuna for its FDR on post-cutoff or uncontaminated code, and producing that number is part of our contribution. We run it with its own threshold procedure, the same confidence score, the same number of samples per problem and the same calibration set size as our method. Beating it means two things:
- realized post-cutoff FDR stays at or below α where selective generation exceeds it;
- coverage stays within a few points of what selective generation achieves in-distribution.
The ablation that isolates the key ingredient keeps everything identical except the rewrite. The calibration set, threshold procedure and confidence score are the same, and only the presence of $R$ changes. A second ablation applies identifier renaming alone, with no change to the narrative, to separate the two components of the rewrite.
Why not just…?
Why not just calibrate on post-cutoff problems? This is the most serious alternative. If enough fresh labeled problems exist, it may simply work, and we treat it as a ceiling rather than a straw man. Its limitations are practical. Fresh problems are few, LiveCodeBench has 612 across fifteen months. They must be relabeled with tests, and they become stale once the next model trains on them, which is the maintenance burden LiveCodeBench itself acknowledges. We run this baseline with small calibration budgets of 50, 100 and 200 problems from an earlier window, evaluated on a later window. If fresh calibration at 100 problems matches our method, we will state that variant anchoring is only useful when fresh data is unavailable, such as for new languages or private domains.
Why not just drop contaminated items with a membership detector before calibrating? Membership scores like PAC from Data Contamination Calibration for Black-box LLMs are imperfect detectors. Its reported gains are about 5 AUC points over prior detectors, so filtering will leave some contaminated items and remove some clean ones. Filtering also shrinks the calibration set and does nothing about contaminated queries at deployment. We include filtered calibration at the same final set size as a baseline. If it matches the rewrite method, the simpler method wins.
Why not just use agreement among sampled programs as confidence? Agreement across samples is a reasonable score, but memorized solutions are exactly the ones a model reproduces consistently. Agreement should therefore be inflated by contamination at least as much as log-probabilities are. We test this in stage two by computing the confidence gap between contaminated and clean halves for each score. Agreement is also an orthogonal choice: if it turns out to be a better score, we apply the rewrite to it as well.
Why not trust that rewriting removes memorization? Detecting Functional Memorization in Code Language Models shows that logic can be memorized behind changed surface text, so rewriting may leave residual inflation. Stage two measures this directly as the confidence gap between contaminated and clean items after rewriting. A large remaining gap would bound how much any surface-level fix can achieve, and we would report it as such.
Proposed Experiments and Decisive Checks
Models. We use StarCoderBase-15.5B, whose pretraining corpus overlap is publicly measured, DeepSeek-Coder-33B and Qwen2.5-Coder-7B, the last flagged as a contamination outlier by DyCodeEval. We add a reasoning model to test whether SALT's high-confidence failures interact with contamination, and the deliberately leaked Llama-3.2-1B for the controlled study. Each problem gets ten samples.
Tests and metrics. Tests are fuzzed suites for the old benchmarks and LiveCodeBench's 18+ tests per problem for new ones. The metrics are realized FDR at α of 0.1 and 0.3, coverage, AUROC, ECE and Brier score.
There are three decisive checks.
The first is whether the selective-generation threshold fitted on HumanEval/MBPP violates its FDR target on post-cutoff LiveCodeBench, and whether the violation is larger for more contaminated models. If it does not, the audit is a negative result, the method is unnecessary, and we report that.
The second, in the controlled leak experiment, is whether contaminated items show higher confidence than clean items at matched correctness, and whether rewriting shrinks that gap. This isolates contamination from difficulty.
The third is whether thresholds fitted on rewritten old benchmarks hold the target FDR on post-cutoff problems at coverage comparable to calibration on 100 fresh problems. A clear yes makes leaked benchmarks reusable for calibration. A no means that only fresh data supports a valid guarantee, which is itself a useful warning for anyone currently deploying thresholds tuned on HumanEval.
References
- Towards Functional Correctness of Code Models with Selective Generation
- Calibration and Correctness of Language Models for Code
- Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
- Data Contamination Calibration for Black-box LLMs
- Detecting Functional Memorization in Code Language Models
- Evaluating LLM Uncertainty in Long-Form Generation Using Deterministic Ground Truth