Verifier Ensembles That Hold Up Under Optimization Pressure in Repository Repair

Coding agents that fix real GitHub issues never see the hidden tests that grade them. They generate several candidate patches and pick one using a stand-in judge: tests they wrote themselves, or a language model trained to predict whether a patch will pass. The trouble is that these judges can be gamed. When an agent keeps editing a patch until the stand-in judge is satisfied, it drifts toward narrow hacks that satisfy the judge and fail the real tests. This is Goodhart's law, the tendency of any measure to break once it becomes a target. Investigating Test Overfitting on SWE-bench measured it directly. The fraction of accepted patches that pass self-generated tests but fail the hidden tests rose from 33.0% to 35.9% for GPT-4o, and from 21.8% to 25.5% for Claude-3.7 Sonnet, after iterative refinement.

We propose to build a panel of judges whose mistakes differ, combine them with weights learned per issue, and retrain those weights on the very patches that fooled the panel in earlier rounds. The weights are trained against a smooth stand-in for the real selection metric. The retraining loop is borrowed from speech enhancement, where a quality-predicting network is kept honest by updating it on the enhancer's latest outputs. The claim is testable. Under repeated refinement pressure, a diverse and adversarially refreshed panel should keep its false-acceptance rate flat while every single judge's rate climbs. The project also contributes a dataset of "judge-fooling" patches for measuring this.

Core Claim

The limiting factor in picking the best repository patch is not how accurate the judge is on patches it was trained on. It is how accurate the judge stays once something optimizes against it. Three things together should resist this gaming. The first is a set of judges with different failure modes: executed generated tests, a language-model reward judge, and cross-patch behavioral agreement. The second is issue-dependent weights over those judges, trained on a differentiable surrogate of "the chosen patch resolves the issue." The third is periodic retraining of the weights on a replay buffer of patches that fooled the previous panel. Each ingredient alone has precedent. The combination, and its evaluation under optimization pressure, does not.

Where This Sits Relative to Prior Work

Two papers motivate the gap. The first is SWE-World, which trains a single language-model reward judge to act as a virtual test runner. It uses that judge to rerank eight candidate patches, reaching 68.2% resolved on SWE-bench Verified with Qwen2.5-Coder-32B. The authors acknowledge that the judge can hallucinate on unfamiliar patches. We keep reranking but replace the single judge with a panel, and we test it under adversarial pressure, which SWE-World does not. The second is Investigating Test Overfitting on SWE-bench, which supplies the measurement protocol, a refinement loop built from Agentless, e-Otter++ and an LLM critic. It studies only single-judge loops. We reuse its loop as the source of optimization pressure.

UTBoost constrains the design. It shows that even the official test suites let through 345 false passes. Its multi-test "intramorphic" check compares behavior against the human reference patch, so it cannot be used at solve time. We use it only as a stricter grader, and borrow its idea of comparing behavior across patches in a form that needs no reference: candidates are compared against each other.

Two papers supply the method. The weighting network follows Differentiable Model Selection for Ensemble Learning, whose method (e2e-CEL) learns input-dependent selection of base models through a smoothed top-k step. It was demonstrated only on image classification, with a fixed number k of selected models. We apply it to choosing among structured code outputs, and let the effective number of trusted judges vary with a temperature. The retraining loop follows the alternating update of MetricGAN+ and iMetricGAN. Those methods refresh a learned quality predictor on current generator outputs plus a history buffer, to stop the generator from exploiting it.

The PESQetarian constrains us in a second way, with a warning. Optimizing a single speech-quality score to 3.82 produced audio with a signal-to-distortion ratio of -19.8 dB. So no single surrogate, however well fit, is trusted alone here.

We also make a deliberate negative choice. We do not backpropagate into the patch text itself. Learning Differentiable Surrogate Losses for Structured Prediction refines small molecular graphs by gradient descent along a learned loss and reports that this landscape is non-convex and traps outputs. Multi-file code patches are a far harder space for that. Gradients therefore flow only into the judge weights.

Research Program

The first component is the dataset. We mine issues with executable tests from SWE-bench-train (19,000 instances, 37 repositories, per SWE-bench) and from the SWE-World issue collection. For each issue we sample eight candidate patches from a fixed open generator. We then run the refinement loop for several rounds against each judge configuration. Every patch is labeled with its hidden-test result and with UTBoost-style augmented tests where the reference patch allows. The patches that pass the judge but fail the hidden tests form the "judge-fooling" set. It is the first collection, to our knowledge, of repository patches produced specifically by pressure against verifiers. Test issues are drawn only from repositories disjoint from training.

The second component is the judge panel. It contains four judges. The first is a set of reproduction tests generated by an e-Otter++-style or SWE-Tester-style model, executed in the container. The second is the repository's visible pass-to-pass regression tests. The third is a language-model reward judge in the style of SWE-World. The fourth is a cross-patch agreement score: how many other candidates' generated tests a patch passes, and how large a cluster of behaviorally equivalent candidates it belongs to.

The weighting network reads issue and patch features and outputs weights $w_m(x)$ for each judge $m$ on issue $x$. Candidate $j$ then receives the score $s_j = \sum_m w_m(x)\, v_m(p_j)$, where $v_m(p_j)$ is judge $m$'s score for patch $p_j$. We train the network on a smooth version of top-1 resolution:

$$ \mathcal{L}(x) = -\log \sum_{j} \frac{\exp(s_j/\tau)}{\sum_{k}\exp(s_k/\tau)}\, r_j , $$

Here $r_j \in \{0,1\}$ says whether patch $j$ passes the hidden tests, and $\tau$ is a temperature that sets how sharply the selection concentrates. As $\tau \to 0$ the loss approaches the non-differentiable "chosen patch resolves the issue" metric. We also test the perturbed top-k variant from e2e-CEL, with a learned per-issue k, for letting the panel choose a behavioral cluster rather than a single patch.

The third component is the alternating loop. In each round, the refiner edits candidates to satisfy the current panel. The resulting patches, labeled by the hidden tests, enter a replay buffer. The weighting network, and optionally the reward judge, is then retrained on the buffer. This is MetricGAN's anti-drift recipe, carried over to a setting where the "generator" is an agent loop that receives no gradients.

Baselines

The naive reference point is the generator's first sample with no selection, together with random choice from the eight-candidate pool. Every other number is read against the gap between the two.

The strongest existing competitor is single-judge reranking with a learned reward model, as in SWE-World. That paper reports 68.2% resolved on SWE-bench Verified at eight candidates with Qwen2.5-Coder-32B. Our generator and pool will differ, so we re-run SWE-World-style reranking on the identical pools, judge-call budget and training issues. Beating it means a higher resolve rate on the same pools at matched judge cost, and more importantly a lower false-acceptance rate after refinement pressure. The second strongest competitor is the single-test-judge refinement loop from Investigating Test Overfitting on SWE-bench, with reported overfitting rates of 35.9% for GPT-4o and 25.5% for Claude-3.7 Sonnet after refinement. We re-run it with our generator so that rates are comparable.

The ensemble baselines use the same panel of four judges. Equal-weight averaging of judge scores and plurality voting over behavioral clusters come first. For reference, Differentiable Model Selection for Ensemble Learning reports that plurality voting reached 80.78% against 90.97% for learned selection on UTKFace, but that result is from image classification and may not transfer. Next comes pointwise stacking: a logistic regression predicting each patch's pass/fail from judge scores, then taking the argmax. All learned methods get the same training issues and the same hyperparameter-search budget.

Two ablations isolate the key ingredients. The first freezes the weighting network after the first round, removing the alternating retraining. The second removes the cross-patch agreement judge, the one judge whose failure mode is structurally different from both test-writing and model-based judging.

Why not just…?

Why not just generate many more reproduction tests and require a patch to pass all of them? Additional tests from the same generator share its misreading of the issue, so they fail together. The overfitting study found that the refinement loop preferred editing the code to fit flawed test assumptions rather than questioning them. Our experiments include a matched-budget arm that spends the whole judge budget on extra generated tests. If that arm matches the panel under pressure, test diversity alone suffices and the panel adds nothing.

Why not just use one stronger reward judge? A single judge's blind spots are exactly what an optimizer finds. PESQetarian-style gaming and SWE-World's acknowledged hallucinations both point this way. This alternative is a real threat on clean pools, and the single judge may tie us there. The decisive measurement is the slope of false-acceptance against refinement rounds, not the resolve rate at round zero.

Why not just average the judges, or stack them with logistic regression? This is the most serious alternative. On unpressured pools it may match the learned listwise weighting, and we will report it if so. Our claim rests instead on the retraining loop, which works equally well with a stacked model. If stacking with replay matches listwise weighting with replay, we will drop the differentiable-selection element from the claim. The contribution then becomes the pressure-tested dataset plus the finding that judge diversity with adversarial refresh is what matters.

Why not just train the generator with reinforcement learning against real execution? Hidden tests do not exist at solve time. Any training signal passes through some stand-in judge, so the gaming question comes back in a different place. The overfitting study also notes that even access to golden tests raised resolved instances by only about 3%.

Proposed Experiments

We evaluate on SWE-bench Verified (500 issues) graded with UTBoost's corrected parser and augmented tests, and on TDD-bench Verified (449 issues) as a second distribution. Training issues come from SWE-bench-train and the SWE-World collection, restricted to repositories disjoint from the test sets. The metrics are top-1 resolve rate on fixed eight-candidate pools, and false-acceptance rate, meaning the fraction of selected patches that fail the hidden or augmented tests. We track false-acceptance across zero to five refinement rounds against each selector.

The decisive check has two parts. After five rounds, the retrained panel's false-acceptance must be statistically flat relative to round zero while the single-judge baselines rise. The panel must also do no worse than SWE-World-style reranking on resolve rate at round zero. A secondary check tests transfer: patches that fooled one judge configuration are used as held-out attacks on the others.

The idea fails if either of two things happens. Equal-weight averaging with replay matches the full method on both metrics. Or the panel's false-acceptance climbs at the same rate as the single judges, which would mean the optimizer finds hacks that fool all four judges at once.