TL;DR
When LLMs repair code after API or runtime migrations, they often produce "silent failures"—code that runs without errors but returns incorrect results. This paper demonstrates that standard testing gates miss a staggering 41% of these wrong repairs, but adding just a single edge-case input derived from release notes can slash that failure rate to roughly 6%.
What Problem Does This Paper Address?
As software ecosystems evolve, developers must migrate code to support new runtimes (like moving from Python 3.9 to 3.12) or updated APIs. While some breaking changes are "loud" (causing immediate crashes), many are "quiet" behavioral changes. For example, a function might now return a more precise float that breaks a ledger total, or an enumeration might render as an integer instead of a string.
LLMs are increasingly used to automate these repairs. However, they suffer from the overfitting problem: a model might produce a patch that passes all existing "typical" tests but fails on edge cases or in production. The authors investigate how often these "accepted-but-wrong" repairs slip through release gates and how we can build better filters to catch them.
Methods
The researchers employed a rigorous, preregistered study design (using SHA-256 hashes to "freeze" their experimental design before running it) to evaluate LLM repairs across four model families (including GPT-5.6-luna and Qwen variants).
Key methodological innovations include:
- Finite Repair Domain: Instead of free-form code generation, models chose from a "menu" of documented code fragments. This allowed the authors to exhaustively evaluate the entire space of possible repairs.
- Release Gates: They compared different levels of testing "gates," ranging from "typical inputs only" to gates supplemented by human-written edge cases, LLM-generated self-tests, and "blind" writer inputs.
- The Blind Writer: They used an isolated frontier model (GPT-5.6-luna) that saw only the release notes and requirements—not the code itself—to generate test inputs. This tested whether a model could "guess" where a migration might fail just by reading the documentation.
- Hard Distractors: In later stages, they intentionally added "wrong-looking-right" code fragments that specifically matched legacy outputs on typical inputs to see if the gates could distinguish them from truly correct repairs.
Datasets
The study utilized a benchmark of 73 tasks covering 60 documented changes across three major open-source lineages:
- CPython: 3.9 → 3.12
- Node.js: 20 → 24
- SQLite: 3.51 → 3.54 (notably, 3.54 was released after the models' training cutoffs).
The repair menu was expanded in stages, eventually reaching 1,491 fragments and including 233 hard distractors.
Key Insights and Ideas
- Inputs Matter More Than Assertions: The study found that a release gate's adequacy depends almost entirely on its inputs, not the complexity of its assertion logic. If you don't have an input that exercises the specific change, no amount of unit testing will save you.
- The "Blind" Advantage: A frontier model that has never seen the code is surprisingly effective at generating bug-revealing inputs just by reading release notes. This "blind writer" performed nearly as well as human experts.
- Self-Tests are a Weak Link: Models are not great at grading their own homework. "Self-tests" (where the repairing model writes its own tests) were significantly less effective than independent "blind" tests, likely because the model's tests shared the same misconceptions as its code repairs.
- Silent Failures are the Norm: Without specific edge-case inputs, nearly half of the "successful" repairs produced by open-weight models were actually incorrect.
Key Results
- Typical Gates are Leaky: Release gates using only typical inputs allowed 41% of incorrect repairs to pass.
- Edge Cases are Powerful: Adding a single author-written edge case reduced the "wrong release" rate from 0.410 to 0.065.
- Frontier Models as Gatekeepers: The "blind" GPT-5.6-luna writer achieved a failure rate of 0.070, proving that AI can be a highly effective automated auditor for migrations.
- Model Scaling: While stronger models (like GPT-5.6-luna) produced fewer wrong repairs overall, they still benefited significantly from edge-case gates. For open-weight models, the improvement was dramatic (e.g., Qwen3-8B went from a 23.3% error rate per attempt to 5.7%).
Limitations and Future Work
- Finite Menu vs. Free-form: The study relies on a menu of code fragments to allow for exhaustive measurement. While this provides scientific clarity, it doesn't perfectly mirror the "infinite" space of free-form coding, though the authors argue the principles of gate adequacy should transfer.
- Sandboxing Constraints: The authors were unable to run a "free-form" arm of the experiment due to strict sandboxing requirements on their study machines, leaving the evaluation of model-written arbitrary code for future work.
- Ecosystem Breadth: The study focused on three specific runtimes. The authors note they make no claims about how these results might translate to web APIs or other software ecosystems.
- AI vs. Human Audit: The contract audit was performed by an AI, not a human panel, which the authors acknowledge as a limitation in resolving certain "class-label" disputes.
Summary updated