Automated program repair tools generate code changes that make a project's failing tests pass. Many of these patches are wrong: they pass the tests without fixing the bug. The field's answer has been a second layer, a learned "judge" that reads the bug report and the patch and predicts whether the patch is actually correct. This proposal argues that such judges can fail in the same way a multimodal medical model fails when it reads the caption and ignores the image. A judge can learn what correct patches usually look like, such as small, tidy, developer-style edits, without checking whether this patch answers this bug. If so, the judge will look accurate on the repair tools it was trained on and break when the patches come from a different source, for example LLM agents on Python repositories instead of search-based tools on Java.
We borrow a diagnostic and a remedy from the cross-modality generalization literature. The diagnostic is a "mirror test" for code. Medical vision-language models drop to chance when an image is flipped, which shows they relied on textbook priors rather than on pixels. The code analogue is to flip a patch's meaning while keeping its surface form, or to pair it with the wrong bug report, and check whether the judge's verdict changes. The remedy is to train the judge on these minimal counterfactual pairs, so that the only way to score well is to model the relation between the report and the code. We test whether this makes judges transfer across patch sources, programming languages, and ecosystems.
Core Claim
Learned patch verifiers are themselves vulnerable to a second-order form of patch overfitting: they reward plausible-looking patches. We expect the effect to be measurable through the judge's sensitivity to counterfactual edits and through how close a diff-only model comes to the full judge. We expect training on report-matched counterfactual pairs to raise cross-source accuracy more than added capacity or added tests do. We claim no new architecture. The contribution is a measurement showing whether today's judges use both inputs, and a training signal that forces them to.
Background and Positioning
Three repair results motivate the problem. Generate-and-validate search spaces contain far more plausible-but-wrong patches than correct ones, and the wrong ones grow faster as the space expands (An Analysis of the Search Spaces for Generate and Validate Patch Generation Systems). LLM repair shows the same gap at scale. On Linux kernel crashes, only about one third of crash-stopping patches fix the root cause (Rethinking Kernel Program Repair: Benchmarking and Enhancing LLMs with RGym). On SWE-bench Verified, roughly one in five "solved" patches from the top-30 agents is semantically incorrect once the tests are strengthened (SWE-ABS). The failure modes are specific: models delete the failing functionality (Automatic Programming: Large Language Models and Beyond), suppress symptoms instead of root causes (FaultLoc), or insert dummy definitions to satisfy checks (TimeMachine-bench). These shortcuts change the patch's meaning more than its appearance, which is exactly what a judge relying on surface form would miss.
The closest prior work, and the method we build on, is Quatrain (Is this Change the Answer to that Problem?). It summarizes each diff into natural language, embeds the summary alongside the bug report, and classifies whether the two match. It reports an AUC of 0.886 on 9,135 Java patches from Defects4J, Bugs.jar, and Bears. Quatrain already shuffles reports at test time and observes a collapse, which shows some reliance on the report. It never asks how much of its accuracy a diff-only model could recover, never applies meaning-flipping edits to the code side, and never evaluates on patches from a different generator or language. We use its data and model as the starting point and change the training signal and the evaluation.
From the cross-modality side, two papers supply the lens. The "Your other Left!" study (Your other Left!) shows GPT-4o falling from 75% to 50% when CT images are flipped. Removing anatomical names in favor of abstract markers raises its accuracy to 87.4%, which shows that a strong text prior can override the evidence. This motivates our mirror test. MER-DG (MER-DG) names "fusion overfitting": when two encoders are trained jointly, each can become lazy and rely on cross-modal shortcuts specific to the training environment. MER-DG counters this by maximizing the entropy of each encoder's representation before fusion. We test MER-DG's regularizer as a secondary arm, not the core mechanism, because it was built for continuous audio and video streams. Its fix targets representation collapse, whereas our hypothesized failure is a missing relation.
Research Program
The first component is the mirror test. For each correct patch (the developer fix), we build two kinds of counterfactual. A semantic flip is a minimal edit that preserves surface form but breaks the fix: negating a guard, swapping a comparison operator, shifting an index by one, or returning the pre-fix value on one branch. Each flip is kept only if it fails the developer's own tests or, failing that, behaves differently from the fix under differential execution, so its incorrect label is checked rather than assumed. A report swap pairs a correct patch with the bug report of a different bug from the same project. For incorrect plausible patches, we form natural minimal pairs with the developer fix for the same bug.
For a judge $f(r, d)$ that scores report $r$ and diff $d$, the reliance score is the average drop in predicted correctness under these interventions:
$$ \rho = \mathbb{E}_{(r,d)\ \text{correct}}\big[f(r,d) - f(r,\tilde d)\big] + \mathbb{E}\big[f(r,d) - f(r',d)\big] $$
Here $\tilde d$ is a semantic flip of $d$, and $r'$ is a swapped report. A judge that models the relation has a large $\rho$. A judge that has learned what patches look like has $\rho$ near zero on flips, even while its AUC is high. Alongside $\rho$, we train report-only and diff-only versions of each judge. The "prior gap" is the joint model's AUC minus the diff-only model's AUC.
The second component is counterfactual training. We retrain the judge with the flipped and swapped pairs added as hard negatives, together with a pairwise ranking loss that requires each true fix to outscore its own flip and its own swapped report. The judge architectures stay fixed: Quatrain itself, a direct code–text cross-encoder without the summarization step (which addresses Quatrain's stated bottleneck), and a prompted LLM judge, which receives the pairs as in-context contrasts or as fine-tuning data. The third component adds MER-DG's marginal and spectral entropy terms to the two encoders of the cross-encoder, which tests whether representation collapse contributes beyond the missing relation.
Baselines
The naive reference point is to accept every plausible patch, which is how test-based benchmarks score today. Its precision equals the fraction of plausible patches that are correct. Applied to SWE-ABS's re-evaluation, that fraction is roughly 80% for top-30 agents. Any judge must beat this precision at a recall of correct patches we fix in advance, otherwise it adds nothing.
The strongest existing learned method is Quatrain (Is this Change the Answer to that Problem?). It reports AUC 0.886 on the combined Defects4J/Bugs.jar/Bears set, filtering 61.7% of incorrect patches at 92.7% recall of correct ones (threshold 0.4). This number is in-distribution. We found no reported result on Lacuna for Quatrain or any learned patch-correctness judge evaluated across generators or languages, on SWE-bench Verified or on kernel patches. We therefore rerun Quatrain from its published setup on our splits. All judges receive the same training patches, the same hyperparameter search budget (equal trial counts on a held-out in-distribution validation split, never on target data), and the same number of training updates. Counterfactual examples count toward the data budget, so the comparison is not just "more data."
The diff-only judge is a deliberate baseline. If it nearly matches Quatrain, the field's reported accuracy comes mostly from patch appearance.
The strongest non-learned competitor is test strengthening. One version is mutation-guided augmentation of regression suites (Are Benchmark Tests Strong Enough?). The other is the adversarial strengthening of SWE-ABS. We run whichever is available on the same target bugs and report the incorrect patches each method catches, alone and combined with the judge.
The ablation that isolates our key ingredient trains on the same total number of extra examples, but the extras are random negatives (incorrect patches from other bugs) rather than minimal pairs matched to a report. If random negatives transfer as well, the counterfactual structure is not what matters.
Beating the field means three things. First, a higher target-domain AUC than Quatrain on both SWE-bench Verified and kernel patches, with non-overlapping bootstrap confidence intervals. Second, more incorrect patches filtered at 90% recall of correct ones. Third, an in-distribution AUC within about one point of the original.
Why not just…?
Why not just generate more tests? This is the most serious threat, because SWE-ABS shows that strengthened tests expose many overfitted patches. Tests can only catch behavior someone thought to check, and generated tests carry the same model biases as the patches. Long et al.'s search-space analysis also shows that more constraints without intent still leave a large pool of plausible wrong patches. We do not argue this point in the abstract. We measure the overlap: if the incorrect patches our judge catches are a subset of those test strengthening catches, the judge is redundant and we will say so. If the two sets are largely disjoint, the methods complement each other.
Why not just use a large LLM as a zero-shot judge? It might be strong. But the "Your other Left!" result shows that frontier models apply strong priors that override the evidence in front of them, and nothing guarantees code is different. We run the mirror test on the zero-shot LLM judge first. If its $\rho$ is high and it already transfers, counterfactual training adds little for that model class, and the contribution shrinks to the diagnostic.
Why not just add the entropy regularizer, or just add capacity? MER-DG's regularizer prevents representation collapse. It does not supply information about which report goes with which diff. Its arm in our design tests this directly: if entropy alone matches counterfactual training, our account of the failure is wrong.
Proposed Experiments
Training uses the 9,135-patch Java set from Quatrain, split by bug so that no bug appears in both train and test. Targets are SWE-bench Verified agent patches, labeled correct or incorrect using SWE-ABS's strengthened evaluation (the labels are reconstructed by rerunning its procedure if they are not released), and kernel patches from RGym, labeled by agreement with the developer root-cause fix as in that paper. Metrics are AUC, the fraction of incorrect patches filtered at 90% recall of correct patches, the reliance score $\rho$, and the prior gap.
The first decisive check is cheap and comes first. We compute the prior gap and $\rho$ for Quatrain and the LLM judge. A small prior gap and low $\rho$ on semantic flips would confirm the core hypothesis before any new training. A large $\rho$ would mean existing judges already use the relation, and we would redirect the work toward the second check alone.
The second check is the cross-source transfer comparison: counterfactual training against Quatrain, random-negative training, the entropy arm, and the LLM judge. Results are broken down by the shortcut categories reported in the repair literature (functionality deletion, symptom suppression, dummy definitions), so we can see which failures each judge misses.
The third check tests whether $\rho$ measured on the source data predicts each judge's target-domain AUC. If it does, the mirror test becomes a cheap pre-deployment check for any patch verifier.
References
- An analysis of the search spaces for generate and validate patch generation systems
- Rethinking Kernel Program Repair: Benchmarking and Enhancing LLMs with RGym
- SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark
- Automatic Programming: Large Language Models and Beyond
- FaultLoc: Evaluating Coding Agents For Fault Localization
- TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks
- Is this Change the Answer to that Problem? Correlating Descriptions of Bug and Code Changes for Evaluating Patch Correctness
- Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images
- MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization
- Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites