Transferable Counterexamples for Code Repair

Language models often write code that passes the visible tests but breaks on inputs nobody thought to try. A natural fix is to search for such breaking inputs and feed them back so the model can repair its code. Existing systems do this one model at a time, and they ask another model to judge which outputs are correct, so that judge's mistakes leak into the feedback. This note borrows an idea from adversarial machine-learning security. There, inputs crafted to fool a small open model often fool unseen commercial models too, because models trained on similar data share similar blind spots.

We ask whether the same holds for bugs. Concretely: is an input that breaks the code of several cheap open-weight models also likely to break the code of an unseen frontier model? If it is, such inputs can be found once, checked against an executable specification instead of a model's opinion, shrunk to their smallest form, and stored in a reusable bank. Any model's repair loop can then draw on that bank.

The opportunity follows from a negative finding. Several papers show that models share misconceptions about code. Self-written tests miss the model's own bugs (Zaili Yang et al.). Model-generated test suites cluster around the narrow error patterns typical of model-written code, the "homogenization trap" described in Rethinking Verification for LLM Code Generation. Prior work treats this correlation purely as a weakness of self-testing. We treat it as a resource: if models make the same mistakes, the bugs of one model are informative about the bugs of another.

Core Claim

The main hypothesis is that bug-exposing inputs transfer across model families. The transfer rate should rise with the number of distinct surrogate families an input breaks, in the same way that ArrAttack keeps only jailbreak prompts that survive perturbation in order to get prompts that work on unseen models.

To make this measurable, let $P$ be a target model's program that passes the public tests. Let $x$ be an input that satisfies the problem's constraints and breaks programs from $k$ distinct surrogate families. Define the conditional transfer rate $T(k) = \Pr[P \text{ fails on } x \mid x \text{ breaks } k \text{ surrogate families}]$. Compare it with the base rate $T_0$, the chance that a random constraint-satisfying input breaks $P$.

The claim is that $T(k) \gg T_0$ and that $T(k)$ increases with $k$. The second part is what would separate shared misconceptions from mere input hardness. The second claim is that a bank of such inputs, labeled by an executable specification and shrunk to minimal size, beats the alternatives as repair feedback for a black-box target. The comparison is at matched target-model compute, against the target attacking itself, against random fuzzing, and against self-debugging. We expect the gain to be largest after the 3–4 round plateau where repair loops normally stall (Yan et al.; AlgoVeri).

The three input directions play different roles. Cross-model transferability is the core mechanism. Iterative refinement is where the bank gets used. The formal verification gap acts as a constraint: we do not trust any model to decide which output is correct, so labels come from machine-checkable specifications.

Relation to Prior Work

The closest work is AdverMCTS. It plays a solver tree search against an attacker tree search, finds inputs on which candidate programs disagree, and relies on a model "Arbiter" to decide the correct output. It reports 72.77% Pass@1 on APPS-Competition. It differs from this proposal in three ways. Its attacker lives inside one model's closed loop, so transfer is never tested. Its labels come from a model judge that can be wrong on exactly the hard cases that matter. And its attacker overfits when given more search, producing brittle tests that lower solver accuracy. We replace the judge with executable specifications and replace per-problem self-play with a filter for inputs that work across model families. We expect the cross-family filter to resist the overfitting problem, because an input tuned to one model's quirks should fail the filter.

Three other works supply pieces of the method.

  • ALGO shows that models write correct slow brute-force reference programs far more reliably than fast solutions: 88.5% of its generated reference programs were semantically correct, and the brute-force success rate exceeded the efficient-solution rate by 822% on CodeContests. Comparing a candidate's outputs against such a reference program is one of our labeling sources.
  • PGS shows that the smallest failing input repairs bugs better than long test traces, reporting 1.4x to 1.6x higher fix rates than prior debugging methods. We reuse its minimization step.
  • AlgoVeri provides 77 classical algorithms with aligned functional contracts, which we compile into runtime checkers.

Four papers motivate the direction but leave the question open:

  • VeriScale uses red-team models to write buggy programs that slip past weak Lean specifications. It is evaluation-only and depends on ground-truth preconditions, and its authors leave closed-loop repair as future work.
  • TestCase-Eval and SAGA build static test suites and never ask whether tests derived from one model's bugs expose another's.
  • DafnyProver feeds the Dafny verifier's counterexamples (concrete variable values for which a proof fails) back to the model. Those counterexamples come from one solver on one program, with no adversary.

Two findings constrain the design:

  • The DafnyProver result that 46 of 52 verified Medium solutions failed runtime limits warns that functional checking ignores efficiency. Our runtime checkers therefore enforce time limits separately.
  • The finding in DecodingTrust that transfer can rest on shared training data cuts both ways. If surrogates and targets share data, transfer may be high, but it may also reflect contamination. We control for this using post-cutoff LiveCodeBench problems.

Research Program

Building the bank. For each problem, we sample programs from four or five open-weight code-model families. We keep the programs that pass the public tests. We then generate inputs that satisfy the problem's constraints using a mix of model-written generators and type-aware mutation, the input-generation method used by VeriScale. We keep inputs on which the surviving programs disagree.

Labeling. Each input's correct output, or its acceptability, is decided by an executable specification. On AlgoVeri tasks, the Dafny contract is compiled into a runtime pre- and postcondition check. On TACO and LiveCodeBench, the check is an ALGO-style brute-force reference program run under a bounded input size. Where neither exists, the problem is excluded rather than handed to a model judge.

Filtering and shrinking. Each labeled breaking input is scored by how many distinct surrogate families it breaks. It is then shrunk PGS-style, by repeatedly deleting and simplifying parts of the input while it still fails and still satisfies the constraints and the specification check.

Analysis. We estimate $T(k)$ for held-out target models: two black-box frontier models and one open model withheld from the surrogate pool. We stratify by algorithm category using TACO's 36 categories, which tests whether transfer is concentrated in specific misconceptions such as off-by-one boundaries, overflow, or empty inputs.

Repair. At each round of a repair loop on the target model, we choose the bank input with the highest $k$ that breaks the target's current program. We feed it back together with its expected output and the program's actual output.

Baselines

The naive reference point is single-pass generation by the target model, scored on hidden tests. Alongside it we run the standard execution-feedback loop: the target sees public-test failures and tracebacks, in the style of Chen et al.. On LiveCodeBench, this kind of feedback raised GPT-4-Turbo from 24.5% to 36.9% in the original paper. Beating it requires a significant gain in hidden-test Pass@1 at the same number of repair rounds.

The strongest competitor is the target attacking itself, in the style of AdverMCTS: the target generates candidates and inputs on which they disagree, and a judge labels them. This is the strongest competitor because it searches the target's own bugs directly, so if anything should outperform transferred inputs, it should. We run it two ways. In the original form, a model judge labels the inputs. In the strengthened form, our executable specification does the labeling, which isolates the value of transfer from the value of better labels. Both versions get the same per-problem target-model token budget as our method.

Surrogate compute is reported separately, both as a one-time cost spread over all targets and as a per-problem cost. The reference number is AdverMCTS's 72.77% Pass@1 on APPS-Competition, though its backbone differs. We rerun the method on our targets rather than comparing against that figure directly.

The third baseline is random fuzzing: random constraint-satisfying inputs checked against the same executable specification, with the same number of inputs supplied per round. Two ablations isolate what we add. One removes the cross-family filter and uses every surrogate-breaking input regardless of $k$. The other removes shrinking.

For test-quality comparisons on TestCase-Eval, the reference is its best model, Qwen3-32B, at 43.8% fault exposure, against 93.3% for human experts. No reported result was found on Lacuna for cross-model transfer of bug-exposing inputs, so there is no prior figure for $T(k)$.

Why not just…?

Why not just let the target attack itself? This is the real threat. A self-attack has direct access to the target's own failure modes. Transfer only wins if self-search is limited by the shared-misconception problem: the target's attacker tends to miss the target's own bugs, as TestCase-Eval and Yang et al. suggest. Transfer also wins if self-search overfits as its budget grows, which AdverMCTS reports. The strengthened self-attack baseline, labeled by our specification, settles this directly. If it matches the bank at equal target-model compute, the remaining contribution reduces to reuse across targets, and the note will say so.

Why not just fuzz against the reference program? If random inputs expose bugs as often as transferred ones, then transfer adds nothing beyond labeling. The comparison of $T(k)$ with $T_0$ tests this, especially on TACO's Hard and Very Hard splits, where GPT-4's Pass@1 falls to 2.0% and bugs plausibly need structured inputs. Brute-force reference programs also limit fuzzing to small inputs. That is why shrinking matters: it keeps transferred counterexamples within the size the reference program can handle.

Why not just use the benchmark's hidden tests or self-debugging? Hidden tests do not exist for new problems, and using them in the loop leaks the evaluation. TACO's roughly 200 tests per problem are reserved strictly for scoring. Self-debugging is already the naive baseline, and the circularity results of Yang et al. predict that it stalls.

Proposed Experiments

The first decisive check is transfer itself. On TACO, APPS-Competition, and post-cutoff LiveCodeBench, we estimate $T(k)$ for $k = 1$ to $4$ against held-out targets and compare it with $T_0$. The proposal fails if $T(k)$ does not clearly exceed $T_0$, or if it does not rise with $k$.

The second check is labeling quality. On AlgoVeri, where executable contracts give exact labels, we measure how often a model judge mislabels contested inputs. This quantifies how much of the error in AdverMCTS-style systems comes from the judge.

The third check is repair. We run a 15-round loop matching the AlgoVeri round budget and plot hidden-test Pass@1 per round for every baseline and ablation. The central prediction is that bank feedback keeps improving after round 4, while self-debugging and self-attack plateau.

The fourth check tests whether the bank generalizes from model bugs to human bugs. On TestCase-Eval's human-written buggy submissions, we measure how many the bank exposes, against the 43.8% model reference. Human errors are a harder target than model errors because they need not share model misconceptions.

Throughout, runtime limits are enforced separately from functional checks. This avoids the gap between verified and executable code seen in DafnyProver.