Counterfactual Test Pairs Against Test Overfitting in Iterative Program Repair

When a language model repairs a bug by repeatedly running a test, reading the failure, and editing the code, it can learn to satisfy that one test instead of fixing the bug. For example, it might special-case the exact input the test uses. This failure is well documented. On SWE-bench Verified, iterative refinement against model-written reproduction tests raised the share of patches that pass the surrogate test but fail the developers' hidden tests. For Claude-3.7 Sonnet the rate went from 21.8% to 25.5%, and for GPT-4o from 33.0% to 35.9% (Is the Cure Still Worse Than the Disease?, Investigating Test Overfitting on SWE-bench). In causal terms, the repair loop treats incidental details of one test as if they were the specification. This is the same shortcut problem that counterfactual data augmentation was built to break. Counterfactual data augmentation means training or evaluating on "what-if" variants of each example, where incidental details change and the meaningful ones are held fixed.

We propose to give the repair loop counterfactual test pairs instead of single tests. Each pair runs the program twice. The two inputs differ only in a factor that the issue implies should not matter, such as a literal value, a variable name, the order of independent operations, or the input size. Alternatively, they differ only in the factor that triggers the bug. The pair asserts a relation between the two runs ("the outputs must match", "the output must scale this way", "only the triggering input should change behaviour") rather than an absolute expected output. The key property is that such relations can be checked without knowing the correct answer. Ordinary generated tests need an oracle (a source of the correct expected output) and are often wrong in ways models blindly trust. A patch that hard-codes the surrogate test's case will usually violate the relation on the paired input.

Core Claim

Relational counterfactual tests, checked by execution before use, cut test overfitting in iterative repair more than the same number of extra ordinary tests does. The benefit should come specifically from replacing absolute assertions with paired, oracle-free relations.

Concretely, a pair consists of a test input $x$, a counterfactual input $x' = g(x)$ produced by an intervention $g$ on one factor, and a relation $h$. The patched program $f$ satisfies the pair when

$$f(x') = h\big(f(x)\big),$$

where $h$ is the identity for nuisance interventions and a stated transformation otherwise. A causal-contrast pair instead asserts that the original buggy code fails on the triggering input and behaves identically to the patched code on its non-triggering twin.

This claim can fail in two ways, and either would be informative. Relations inferred from issue text might be wrong so often that they reject correct patches. Or plain test diversity might explain all of the gain.

How This Differs From the Closest Work

Three pieces of prior work bound this proposal.

The first motivates it. Ahmed et al. measure overfitting inside a refinement loop of up to 15 iterations. That loop uses Agentless for patching, e-Otter++ for reproduction tests, and a critic that sees execution logs and coverage. They show that hiding logs does not help and that even access to the golden tests leaves 5.8% (Claude) and 11.3% (GPT-4o) regression-breaking overfitting. They also find correct patches have higher mean coverage than overfitted ones (0.95 versus 0.90). However, they propose no intervention on the tests themselves. We supply one.

The second two supply the method but in other settings. iFlip refines counterfactual text iteratively, using a classifier's feedback and early stopping, for sentiment, topic, and inference tasks. VISION generates paired vulnerable/fixed C functions in a single LLM pass, never compiled or run, to train a graph-based vulnerability detector. We borrow iFlip's closed loop and VISION's minimal-pair idea. Our pairs, however, are executable tests that act as feedback and as a filter inside a repair loop, not training data for a classifier. Their validity is checked by running them.

The causal justification follows Selecting Data Augmentation for Simulating Interventions, which treats augmentations as simulated interventions, and Explaining the Efficacy of Counterfactually Augmented Data, which argues that paired data removes any reward for fitting nuisance features. Here the "learner" is a search over patches rather than a gradient-trained model, so the argument transfers only by analogy. We test that analogy rather than assume it.

CRCA-Context applies counterfactual rewordings to issue descriptions and finds that file retrieval is fragile to them. That work perturbs the input to retrieval. We perturb the inputs to the program and leave retrieval untouched.

Research Program

The system has three parts.

The first part is a pair generator. Given the issue, the surrogate reproduction test, and the relevant source, an LLM proposes named factors, labels each as "nuisance" or "trigger", and writes pairs with relational assertions.

The second part is a validation loop in the style of iFlip, with an executor in place of a classifier. Every pair must import and run on the original repository without setup errors. Each nuisance relation must hold on the unpatched code for inputs outside the issue's triggering region, so the relation is not already contradicted by existing correct behaviour. Each trigger pair must reproduce the failure on the unpatched code. Pairs that fail these checks go back to the generator with the execution trace, for up to a small fixed number of rounds, with early stopping; iFlip reports that forcing extra rounds degrades output. Pairs that still fail are discarded.

The third part integrates the validated pairs into the repair loop of Ahmed et al. They are added as extra feedback during refinement, and they act as an acceptance gate: a patch is submitted only if it passes the surrogate test and all validated pairs, with rollback to the last passing patch otherwise.

The analysis looks at which intervention types catch which overfitting patterns. We will tally hard-coded literals, narrowed conditionals, and edits confined to the test's code path. We will also measure how often validated relations reject the developer's golden patch, which is a direct false-alarm rate. The golden patch is used only for this offline analysis, never at test time.

Baselines

The naive reference is the refinement loop of Ahmed et al. as published: Agentless patches, e-Otter++ surrogate tests, and critic-guided refinement. Its reported overfitting rates are 25.5% for Claude-3.7 Sonnet and 35.9% for GPT-4o after refinement, versus 21.8% and 33.0% with no refinement. We rerun it ourselves on the same 449 TDD-bench Verified instances, with the same models and iteration cap, so all comparisons share data and sampling budget. The golden-test-access condition (5.8% and 11.3% overfitting) serves as an unreachable-in-practice ceiling reference.

The strongest practical competitor is the same loop augmented with more ordinary tests at a matched budget. We sample additional e-Otter++-style tests until their count and total LLM tokens equal those spent on generating and validating pairs. Passing all of them is required, and the existing repository test suite also runs in the loop. This is the baseline most likely to erase our gain, because diversity alone may catch hard-coded patches.

We also include coverage-guided selection, which picks the final patch among candidates by the coverage signal Ahmed et al. found separates correct from overfitted patches. No reported selection accuracy for coverage was found on Lacuna, so we measure it.

The key ablation keeps our generator and validation loop but converts every pair into two absolute-assertion tests, with expected outputs written by the LLM. This isolates the oracle-free relation as the ingredient.

A second ablation removes the validation loop, giving open-loop pairs as in VISION. This isolates the closed-loop verification.

Beating the matched-budget baseline means lower overfitting at an equal or higher net number of resolved instances, with the difference exceeding the variation across seeds. Net resolved is resolved instances minus instances broken that were previously working, the accounting Ahmed et al. use to show a nominal +8 was really +5.

For context only, BOOSTAPR reports a 40.7% resolve rate on SWE-bench Verified with reinforcement learning. That result comes from a different training regime and does not report overfitting, so it is not a like-for-like target.

Why not just…?

Why not just hide the execution logs so the model cannot game them? Ahmed et al. tested this directly, giving the model only pass/fail flags, and overfitting "remained stubbornly high." Removing tracebacks also cripples legitimate self-correction, as in Hollmann et al.. The shortcut lives in the single test's content, not in the verbosity of the feedback.

Why not just generate many more ordinary tests? This is the real threat. If broad sampling of e-Otter++-style tests matches our results at equal cost, the counterfactual framing adds nothing, and we would report that. Our reason to expect a gap is that every extra ordinary test brings its own guessed expected output. Ahmed et al. show models trust flawed tests and edit code to satisfy them. Relational pairs avoid guessing outputs entirely. The matched-budget baseline and the absolute-assertion ablation settle this question together.

Why not just select patches by code coverage? Coverage differs between correct and overfitted patches only on average (0.95 versus 0.90), and nothing in the file shows it discriminates individual patches. We compare coverage and pair consistency as detectors of overfitting on the same pool of patches, and we also test their combination. If coverage alone matches pair consistency, the simpler signal wins.

Why not just run the repository's existing test suite in the loop? Existing suites catch regressions, but they do not test the new behaviour the issue asks for. Even golden tests, a strictly stronger oracle, leave 5.8–11.3% overfitting. We therefore include the existing suite in every baseline, and the question becomes whether pairs add value on top of it.

Decisive Experiments

The main experiment runs every condition on the 449 TDD-bench Verified instances with Claude-3.7 Sonnet and GPT-4o, over three seeds. It reports overfitting rate, net resolved instances, and the number of previously-working instances broken.

The detection experiment scores a fixed pool of patches, labelled correct or overfitted by golden tests. It compares pair consistency, coverage, and matched extra tests by area under the ROC curve, for which no prior result was found on Lacuna.

The false-alarm experiment measures how often validated relations reject golden patches. A rate high enough to cost more resolved instances than overfitting saves would falsify the approach.

As a transfer check, we repeat the main comparison on Defects4J v2.0, where BOOSTAPR reports 24.8% resolved. The pair generator would then face Java and different bug types.

The proposal succeeds if relational pairs beat the matched-budget baseline and the absolute-assertion ablation on overfitting without losing net resolved instances. It fails, instructively, if the gain vanishes once test count is matched.