A code benchmark grades a model by running its programs against a handful of unit tests. Many programs pass those tests and are still wrong. When the hidden bug is found later, the reported score turns out to have been inflated by these "false passes." There are two ways to repair a leaderboard today. One is to rebuild the tests for every problem at large scale, as the "plus" suites of HumanEval and MBPP do. The other is to run an expensive bug-hunting verifier on every passing program. Both costs grow with the number of models times the number of passes.
This note proposes a third option. Learn a cheap predictor of which passes are false. Then spend expensive verification only on a small random audit, chosen so that the final corrected score comes with a valid confidence interval whether or not the predictor is any good. A better predictor only makes the interval narrower. The audit design comes from distribution property testing, the "learn-then-test" pattern: first learn an explicit candidate model from a few samples, then check it against fresh samples. One lesson from that theory, that the candidate must never assign near-zero probability to any outcome, becomes a concrete rule for how audit samples are allocated. The work also tests empirically whether that lesson matters at realistic sample sizes. The theory papers it comes from report no experiments of their own.
Core Claim
For any benchmark, model, and pool of passing programs, we can estimate the false-pass-corrected score to within a stated tolerance, with finite-sample coverage. We claim this takes a number of expensive verifications well below the number of passes, and well below what a uniform random audit would need when the cheap predictor is informative. The design has three pieces. A learned predictor $f_j \in [0,1]$ estimates the probability that pass $j$ is false, using cheap signals: coverage, mutation survival, disagreement among sampled solutions, and violations of model-written properties. A sampling rule audits pass $j$ with probability $\pi_j$. The rule is weighted toward passes where the predictor is uncertain, and it is bounded below by a floor $\pi_{\min}$. Finally, a corrected estimate of the number of false passes is formed:
$$\hat\theta = \sum_{j=1}^{N} f_j + \sum_{j \in S} \frac{Y_j - f_j}{\pi_j},$$
Here $N$ is the number of passes, $S$ is the audited set, and $Y_j \in \{0,1\}$ is the expensive verifier's verdict on whether pass $j$ is false. The estimate is unbiased for any predictor $f$. Its variance scales with $\sum_j (Y_j - f_j)^2 (1-\pi_j)/\pi_j$. The $1/\pi_j$ factor plays the same role as the chi-squared distance in learn-then-test property testing. Small allocation probabilities on passes the predictor gets wrong blow up the error, just as a candidate distribution that underestimates some outcome's mass blows up a chi-squared identity test. The floor serves two purposes. It bounds every importance weight by $1/\pi_{\min}$. That bound is what makes a finite-sample Bernstein-type interval valid rather than only an asymptotic one.
Where This Sits in Prior Work
Two papers motivate the problem. The Limits of Inference Scaling Through Resampling shows that sparse unit tests let weaker models' wrong programs through at high rates. It also shows that this caps what resampling can achieve. It measures false passes by re-running everything against HumanEval+ and MBPP+, but it does not ask how to get that correction without exhaustive re-testing. Position: Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility audits 672 benchmarks and finds that 85.0% do not ensure reliable judgment. Over the same period benchmark creation surged, with 210 benchmarks in 2024 and 316 in 2025. Exhaustive repair cannot keep up with that pace, so a cheaper certification method has to be the answer.
Several works supply the expensive and cheap verifiers that we reuse, not compete with. CodeHacker generates targeted hack inputs, and Scaling Agentic Verifier for Competitive Coding trains an agent to find inputs on which candidate programs disagree. Both find bugs effectively, but the agentic verifier costs about 45.5 GPU-seconds per query. That cost is exactly why we use it only on an audit sample. Use Property-Based Testing to Bridge LLM Code Generation and Validation shows that models write valid properties far more often than correct code on hard tasks: 48.9% versus a 1.1% direct pass rate on hard LiveCodeBench problems. That makes property violations a strong cheap feature. The same paper flags property hallucination as a risk, which is why we never trust the predictor without an audit.
The statistical design draws on three lines of work. Optimal Testing for Properties of Distributions and A Framework for Testing Properties of Discrete Distributions develop learn-then-test with chi-squared learning and support trimming. These are purely theoretical. We borrow the structural lessons and test whether they survive at finite sizes. PAC Verification of Statistical Algorithms proves that checking a claimed result can need far fewer samples than producing it. Our audit is an empirical analogue: checking a predictor's corrected leaderboard should need fewer expensive calls than computing it from scratch. Learn then Test calibrates model parameters with family-wise error control and finds classical uniform bounds too conservative in practice. We reuse its fixed-sequence testing for multi-model ranking claims. The object being calibrated differs, though. That paper calibrates a predictor's threshold; we certify a benchmark's score.
Research Program
We first build the cheap predictor. It is a small gradient-boosted classifier, or a lightweight model-based judge, trained on a held-out split of passes with known verdicts. Its features include line and branch coverage of the passing tests, mutation-survival counts, how many alternative sampled solutions disagree with the candidate on random inputs, and property violations produced by a PGS-style tester. The audit allocator then sets inclusion probabilities proportional to the predictor's estimated error spread, $\sqrt{f_j(1-f_j)}$, subject to the floor and to an expected budget $m = \sum_j \pi_j$. This is the standard allocation that minimizes variance, adjusted by the floor.
The estimator yields per-model corrected scores, each with an interval. Pairwise ranking claims are tested in a fixed order with family-wise error control. The finite-sample interval comes from an empirical-Bernstein bound on the bounded weighted terms.
Three empirical questions drive the analysis. The first is whether the floor, the chi-squared lesson, actually changes coverage and width at budgets of a few hundred audits, or whether unfloored allocation is fine in practice. The second is how interval width trades off against predictor quality, measured by correlation with the true verdict. The third is whether a predictor trained on one benchmark keeps its value on a newer benchmark. LiveCodeBench documents a contamination cliff after training cutoffs, which suggests the predictor may drift, and the audit's validity guarantee must hold regardless.
Baselines
The naive reference point is to report the raw pass rate under the benchmark's own tests, with no correction. It costs nothing and is biased upward by the false-pass rate. Any method we propose must state an interval that contains the true corrected score where the raw score does not.
The gold reference is full re-verification of every pass with the expensive verifier. On HumanEval and MBPP this means the plus suites; on competitive-programming sets it means CodeHacker-style hacks or the agentic verifier. Its cost sets the denominator of our savings. Using the cited per-query figure of about 45.5 GPU-seconds from Scaling Agentic Verifier for Competitive Coding, we report how many GPU-hours our audit saves for each leaderboard.
The strongest competitor on the actual question is a uniform random audit with an exact binomial interval: sample $m$ passes uniformly, verify them, and use Clopper–Pearson bounds. It is simple and valid, and it needs no predictor. Every comparison uses the same expected budget $m$ and the same expensive verifier. Our method beats it if, at 95% nominal coverage with empirical coverage of at least 94% over 1,000 simulated audits, it reaches the same interval width with at most half the expensive calls on at least two benchmarks. We found no reported result on Lacuna for audited or certified false-pass-corrected scores. The closest reported numbers are the gaps the plus suites expose. For example, DeepSeek-Coder-33B sits only 4.3 points behind GPT-4 on HumanEval+ but 16.2 points behind on LiveCodeBench code generation, per LiveCodeBench.
The fourth baseline is the cheap predictor used alone, with no audit: PGS property checks as a filter, or the learned classifier thresholded. PGS reports 89.0% pass@1 on HumanEval with DeepSeek-Coder-V2, up from 76.2% with direct prompting, in Use Property-Based Testing to Bridge LLM Code Generation and Validation. We expect a predictor-only correction to be biased, and we measure its coverage directly; beating it means valid coverage where it fails.
Two ablations isolate the key ingredients. In the first, predictor-weighted allocation runs without the floor, which tests the chi-squared lesson. In the second, the predictor is used in the estimator but allocation stays uniform, which separates the gain from the control term from the gain from targeted sampling.
Why not just…?
Why not just expand every benchmark's tests, as HumanEval+ did? Where this is affordable, it is the right answer, and on HumanEval-sized benchmarks it is a real threat to the proposal's usefulness. The case for auditing rests on cost and fragility at scale. CodeHacker shows that large generated suites inherit bypass and rejection errors from flawed checkers. The audit position of Position: Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility shows that most benchmarks never invest in reliable judgment at all. We report the break-even budget at which full re-verification becomes cheaper than auditing, so a reader can see in which regime each approach wins.
Why not just audit a uniform random sample? This is the most serious alternative. If the cheap predictor correlates weakly with true verdicts, our variance gain disappears and uniform auditing is just as good. The experiments settle this directly by plotting width savings against measured predictor correlation. If the savings stay under about 1.5× on every benchmark, the conclusion is that uniform auditing suffices and the learned component is not worth it, and we would report exactly that.
Why not just trust a property-based or model-based checker? Use Property-Based Testing to Bridge LLM Code Generation and Validation itself names property hallucination as a failure mode. The Limits of Inference Scaling Through Resampling shows that verifier errors fall unevenly across models, and that unevenness is precisely what reorders a leaderboard. The predictor-only baseline measures the resulting coverage failure model by model.
Why not require formal proofs instead of tests? VerifyThisBench reports average zero-shot pass rates of 1.48%, rising to 6.22% after refinement. At those rates, formal proofs cannot score today's code leaderboards.
Proposed Experiments
The first stage runs where full ground truth is known: HumanEval versus HumanEval+ and MBPP versus MBPP+, on passes from several open and closed models. Because every verdict is known, we can replay thousands of simulated audits at budgets from 50 to 1,000. From these we measure empirical coverage, interval width, and whether pairwise rankings are correct, for every method and ablation.
The second stage moves to LiveCodeBench and CodeHackerBench. The expensive verifier is a CodeHacker-style hack generator or the agentic verifier, and we treat exhaustive verification on a held-out subset as ground truth.
The third stage is a drift test. We train the predictor on problems published before a model's training cutoff and audit problems published after it. This probes the case where the contamination cliff should degrade the predictor but must not break coverage.
Three checks decide the outcome. First, coverage stays at or above 94% at nominal 95% in every condition, including under drift; failure here sinks the method. Second, the floored allocator matches or beats the unfloored one in worst-case coverage at small budgets; this is the finite-sample test of the chi-squared lesson. Third, expensive calls fall by at least half relative to uniform auditing at equal width; failure here means the honest recommendation is uniform audits.
References
- The Limits of Inference Scaling Through Resampling
- Position: Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
- CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions
- Scaling Agentic Verifier for Competitive Coding
- Use Property-Based Testing to Bridge LLM Code Generation and Validation
- Optimal Testing for Properties of Distributions
- A Framework for Testing Properties of Discrete Distributions: Monotonicity, Independence, and More
- PAC Verification of Statistical Algorithms
- Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- VerifyThisBench: Generating Code, Specifications, and Proofs All at Once