When a language model fixes code iteratively, it needs a signal that tells it whether the latest edit is better. In realistic settings the only available signal is usually a test the model wrote itself. Optimizing against that test is exactly what goes wrong: the model learns to satisfy the test rather than fix the bug. We propose replacing "does my patch pass my test" with a reward computed from many independently written tests and many candidate patches. Each test is paid according to how well its pass/fail verdicts agree with an independent peer's verdicts on the same patches, after subtracting the agreement expected by chance. Each patch is paid according to how it fares on other agents' tests, weighted by those scores. Tests that pass or fail everything earn nothing, and an agent never grades its own patch.
The approach borrows from peer prediction, a branch of mechanism design that pays agents for reports when the truth can never be checked, using only the statistical correlation between independent reports. Peer prediction has so far been applied to multiple-choice answers and prose reviews. Code is a better home for it. In code, a "report" is a test, the comparison between reports is done by actually executing them, and the resulting matrix of verdicts is cheap and cannot be faked with persuasive wording. The main bet is that this reward matters most when models are trained or searched against it, which is precisely the regime where a naive pass-rate reward gets gamed.
Core Claim
The evidence that motivates this work comes from Investigating Test Overfitting on SWE-bench. That study had agents refine patches against reproduction tests they wrote themselves. On SWE-bench Verified, refinement raised the overfitting rate, meaning the share of patches that pass the generated test but fail the hidden "golden" tests, from 21.8% to 25.5% for Claude-3.7 Sonnet and from 33.0% to 35.9% for GPT-4o. Two further findings constrain the design. Hiding the test code and returning only a pass/fail flag did not help. Even with access to the golden tests, resolved instances rose only by about 3%.
PGS names the underlying problem. When one model writes both the code and the tests, it builds the same misunderstanding into both. PGS answers this by splitting generation from testing, but it treats the separate tester as a truthful oracle. Nothing in PGS stops the tester from producing trivial or wrong properties.
Our claim is that an execution-grounded peer-prediction score gives a reward for tests and patches that needs no ground truth and that keeps tracking golden-test quality under optimization pressure. Under the same pressure, raw pass rate against generated tests degrades. The claim is specifically about the training and search regime. At inference time, a simpler holdout scheme may already capture most of the benefit, and we test that directly.
The Mechanism
Take one task with $M$ candidate patches. The patches come from parallel refinement trajectories, and the tests come from $K$ test-writing agents drawn from different model families. Running every test on every patch yields verdicts $v_{tj} \in \{0,1\}$, where $v_{tj}=1$ means test $t$ passes on patch $j$.
For a test $t$, pick a test $t'$ written by a different agent. Test $t$ is scored by how often the two agree on the same patch, minus how often they agree on two different patches:
$$S(t) = \Pr_{j}\left[v_{tj} = v_{t'j}\right] - \Pr_{j \neq k}\left[v_{tj} = v_{t'k}\right].$$
The second term is the agreement that the two tests' overall pass rates alone would produce. It is the code analogue of the discounting step in GSPPM, which conditions on a generic synopsis so that a review earns credit only for information beyond what everyone would say. A test that always passes or always fails scores exactly zero, whatever its peers do. A test earns a positive score only when it splits the candidate pool in the same way as an independently written test.
Each patch $p_j$ then receives
$$R(p_j) = \frac{\sum_{t \notin T(j)} w_t\, v_{tj}}{\sum_{t \notin T(j)} w_t}, \qquad w_t = \max(S(t), 0),$$
where $T(j)$ is the set of tests written by the trajectory that produced $p_j$. Excluding $T(j)$ is the leave-one-out step. It is similar in spirit to WALLA's peer baseline in Decentralized Aggregation of LLM Predictions via Wagering Mechanisms, but it needs no outcome to be revealed later, which WALLA does need in order to settle wagers.
We use these two quantities in two ways. During refinement at test time, $R$ decides whether an edit is accepted and when the loop stops. During training, $S$ is the reward for test writers and $R$ is the reward for patch writers, both turned into preference pairs for Direct Preference Optimization (DPO), a method that fine-tunes a model directly from pairs of preferred and rejected outputs. This follows how Truthfulness Despite Weak Supervision turns peer-prediction scores into DPO training signal on reasoning questions.
That paper supplies the method template: Source, Target and Expert roles, with the score measuring how much one participant's answer helps a weak model predict another's. It also supplies the warning that peer-prediction guarantees cover only one agent deviating at a time, so groups of agents that coordinate on uninformative answers can still be paid. Our design differs in two ways. It replaces log-probability prediction with executed verdicts. It also places the information in actions, namely tests and patches, rather than in text, which removes the stylistic channel through which LLM judges are swayed.
There is a known weakness we do not hide. If every test writer shares the same wrong reading of the specification, their tests will agree on the wrong behavior and score well. The mechanism's justification rests on writers' errors being roughly independent once the true behavior is fixed. We therefore build the writer pool from different model families. We also test, as an optional extension, prediction reports in the style of the Robust Bayesian Truth Serum. In that extension each writer also predicts what fraction of peer tests a patch will fail, and a patch whose actual failure rate exceeds the predicted rate (a "surprisingly popular" outcome) is flagged as evidence that a shared bias is being resisted. Because that variant works with as few as three agents, it fits the small writer pools we can afford.
Research Program
The work has three stages.
The first stage validates the score itself, with no training involved. On tasks that have golden tests, we measure the true discriminative value of each generated test: the fraction of correct and incorrect patches, as judged by the golden tests, that the test classifies correctly. We then ask whether $S(t)$ ranks tests by this quantity better than simpler proxies, namely raw pass rate, agreement with the majority, and an LLM judge's rating.
The second stage applies the scores to inference-time refinement, using $R$ as the acceptance and stopping rule inside a loop. The loop rolls back rejected edits, following the approach of Fang et al., and caps the number of debugging rounds near the two-round optimum reported by S*.
The third stage, the central one, uses the scores for training. We train test writers and patch writers with DPO on the peer-prediction rewards. A matched control trains them on the gameable reward, which pays tests for failing a buggy baseline and patches for passing their own tests. We track whether each reward drifts away from golden-test quality as training proceeds.
Baselines
The naive reference point is self-refinement against the model's own reproduction test, as studied in Investigating Test Overfitting on SWE-bench. Its reported SWE-bench Verified overfitting rates after refinement are 25.5% for Claude-3.7 Sonnet and 35.9% for GPT-4o. We rerun this baseline with our base models so that numbers are compared like for like. Beating it means a lower overfitting rate at an equal or higher resolved rate.
The strongest existing refinement system in the file is Adversarial Review. It uses a reviewer and a critic who must reach consensus under required disagreement tags and line citations before the coder edits anything. It reports a 75.2% pass rate on SWE-bench Verified and 87% on LiveCodeBench. It is the strongest competitor because it targets the same false-consensus failure through prompted structure rather than through incentives. We run it with the same base models, the same number of model calls per task and the same tuning budget. Beating it means a higher resolved rate at matched calls, or the same resolved rate with fewer calls.
The strongest selection baseline is S*. S clusters candidate programs by execution behavior and generates inputs that distinguish between candidates, reporting that DeepSeek-R1-Distill-Qwen-32B improves from 69.1% to 85.7% Pass@1 on LiveCodeBench. We give S exactly the same candidate pool and the same execution budget as our method.
Two ablations isolate the key ingredient. The first, unweighted holdout, keeps the leave-one-out exclusion but sets all $w_t = 1$. It isolates the chance-corrected scoring. The second, same-family pool, draws all test writers from a single model family. It measures how much the mechanism depends on independent errors.
Why not just…?
Why not just let one model debug from compiler and runtime traces? That signal saturates quickly. In CodeTransOcean, three rounds of self-debugging moved success only from 48.57% to 52.57%. More importantly, compilers do not see logical bugs that run without crashing, and those are exactly the bugs behind test overfitting. We include trace-only refinement as a floor.
Why not just hold out tests written by other models and count passes? This is the real threat. At inference time it may match our method, because exclusion alone removes the most direct way to game one's own test. The unweighted-holdout ablation settles the question. Our prediction is that the gap is small at inference and large in training. Under training, a pass-count reward lets test writers inflate scores by producing trivially passing tests and lets patch writers learn to satisfy the dominant test style. The scoring rule $S$ pays nothing for trivial tests. If the gap stays small in training too, we will report that the useful ingredient is cross-model holdout, not peer prediction.
Why not just cluster by execution behavior and take the majority, as S* does? Majority clustering picks the most common behavior. That is the wrong choice exactly when most candidates share a misunderstanding, which is the self-deception failure PGS describes. We report results separately on tasks where the golden-correct behavior is a minority among candidates, the regime where the two methods should diverge.
Why not just use a strong LLM judge or a prompted adversarial review? Truthfulness Despite Weak Supervision found that LLM judges fell below random accuracy once the participant was 5–20 times larger than the judge, while peer prediction still worked at a 100-fold size gap. Adversarial Review counts as a strong competitor here, and the matched comparison above decides how it fares against our method.
Proposed Experiments
We evaluate on SWE-bench Verified and TDD-bench Verified, using golden tests only for scoring, and on LiveCodeBench. Training uses older LiveCodeBench windows and CodeContests problems, where execution is cheap, and evaluation uses later contamination-free LiveCodeBench windows. We then test transfer to SWE-bench Verified. The metrics are resolved or Pass@1 rate, overfitting rate as defined in the test-overfitting study, rank correlation between $S(t)$ and each test's golden discriminative value, and the fraction of trivial tests that writers produce during training.
The decisive check is the training curve. Under the naive reward, golden-test quality should rise and then fall while the reward keeps climbing. Under the peer-prediction reward, the reward and golden-test quality should keep moving together. If they diverge for both rewards, the mechanism fails in the regime where it was supposed to matter. The second check is the same-family ablation, which shows whether shared bias is the mechanism's limiting failure, as we expect.
References
- Investigating Test Overfitting on SWE-bench
- PGS: Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback
- Eliciting Informative Text Evaluations with Large Language Models
- Decentralized Aggregation of LLM Predictions via Wagering Mechanisms
- Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction
- A Robust Bayesian Truth Serum for Small Populations
- MEIC: Re-thinking RTL Debug Automation using LLMs
- S*: Test Time Scaling for Code Generation
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation