Coding agents are scored on one bit: does the patch pass the repository's tests? That bit is unreliable, and the reason is well understood in software engineering. Many regressions are change interaction errors. These are defects that appear not because the edited lines are wrong on their own, but because a new write to some piece of state reaches an old, unedited line that reads that state and assumed something the patch no longer guarantees. UTBoost found 345 patches on SWE-bench Lite and Verified that passed the official tests but were actually wrong. Ahmed et al. found that agents refining against their own tests break functionality elsewhere more often, not less. Neither line of work asks which mechanism produced the hidden breakage, or at what stage of understanding the agent failed.
We propose to build, for every SWE-bench task, a short ladder of execution-checked questions about the change. Can the model find the lines that write state? Can it name the unedited lines that read that state (the "altered write-read pairs")? Can it predict how behavior at those reads will change? Only then do we ask whether it can produce a patch that keeps those reads correct. Every answer is graded against a dynamic trace of real executions, not by an LLM judge. The central bet is that the middle rungs, especially naming the affected unedited reads, predict hidden regressions and false passes much better than test pass rate or patch complexity does. The same tracer, applied to a candidate patch without any reference solution, should also give a cheap way to flag those regressions.
Core Claim
A large share of false passes and overfitted patches on SWE-bench are change interaction errors located at altered write-read pairs in unmodified code. An agent's failure at the rung "identify the unmodified reads affected by this change" predicts these hidden failures over and above flat pass rate and patch complexity. What is new relative to prior work has three parts. The hierarchy is defined over the program's dependency structure, not over the agent's reasoning steps. The ground truth at every rung comes from execution rather than a judge. And the hierarchy is tested for predictive validity, meaning whether it forecasts real downstream breakage, rather than only for descriptive coverage.
What Prior Work Gives Us and Where It Stops
The mechanism comes from the change-interaction literature. Regression tests to expose change interaction errors builds change sequence graphs (maps of how edited statements can reach each other through control and data dependence) and uses directed symbolic execution over them. In GNU Coreutils it exposed all change interaction errors, while testing changes in isolation exposed about 57%. Mitigating Security Risks in Linux with KLAUS formalized the altered write-read pair and fuzzed toward "modified write, then unmodified read," finding 30 incorrect patches among 250 Linux patches. On the use of commit-relevant mutants showed that the code a commit actually affects mostly lies on unmodified lines, and that targeting it gives a 30% higher chance of revealing commit-introduced bugs at the same budget. We reuse these ideas as method. We also inherit their limits: symbolic execution does not scale to large Python repositories, and KLAUS is specialized to kernel memory errors. For that reason we use lightweight dynamic def-use tracing over existing test executions.
Two findings constrain the design. CoREBench reports that change interaction errors have mean cyclomatic change complexity of 20.1 versus 9.9 for other regressions. A plain complexity score might therefore explain everything our ladder claims to explain, and we treat it as a primary competitor. The hierarchical-evaluation literature also warns that ladders are not monotone: models can be right at the top while wrong in the middle. We measure this "ungrounded correctness" directly rather than assume it away.
On the evaluation side, AgentEval models agent trajectories as graphs of steps with a three-level failure taxonomy and an LLM judge. Step-level evaluation raises failure-detection recall from 0.41 to 0.89. Its graph, however, is the agent's own workflow, and it never inspects the data flow of the repository being edited. ToolBeHonest shows how a three-level diagnostic depth reveals failures that flat scores hide, but only for tool calling. SynFix is the closest repair system. It builds a repository graph of imports, calls and inheritance, and has the model check one-hop neighbors of each edited entity for needed coordinated edits. It reaches 55.8% on SWE-bench Verified, but it uses syntactic one-hop structure and is graded flat. UTBoost and Investigating Test Overfitting on SWE-bench supply our outcome labels: false passes and overfitted patches. Both treat the failure as a black box, and UTBoost needs the human gold patch as an oracle.
Research Program
The first component is a dynamic interaction tracer for Python repositories. For a pre-patch version, a patch, and the repository's test suite, it runs the tests under an execution tracer. It records writes to variables, attributes, and container contents that originate on patched lines, and every later read of the same storage location that occurs on unpatched lines. Each such pair is an altered write-read pair. We record its hop distance in SynFix-style structural terms (same function, one-hop neighbor, or farther), so that the one-hop restriction can be tested directly. We accept that the tracer only sees behavior that existing tests exercise, and we report that coverage limit per task.
The second component is the ladder. For each task in SWE-bench Verified, the tracer runs on the gold patch to produce ground truth for four rungs. On the first rung, the model is given the issue and repository and asked to locate the lines whose writes must change. These are scored against the gold patch's write sites. On the second rung, the model is given those write sites and asked to name the unmodified locations whose reads will see different values. These are scored by precision and recall against the traced pairs. On the third rung, the model is given a specific pair and a test input, and asked to predict whether the value observed at the read changes and how. This is scored by execution. The fourth rung is ordinary patch generation, scored with the original tests plus UTBoost-style augmented tests. Each rung gives the model the correct answer to the rung below. A failure at rung three therefore cannot be blamed on a rung-two miss, and we can report where capability actually breaks. We separately record cases where rung four succeeds despite rung-two failure, which is the ungrounded-correctness rate.
The third component is an oracle-free detector. It runs the same tracer on a candidate patch: it executes existing tests before and after the patch, and flags pairs where the value at an unmodified read changed. It then asks a model whether each divergence is explained by the issue text. This is Böhme-style differential testing aimed by altered write-read pairs and needs no gold patch, so it could be used in deployment rather than only on historical benchmarks.
Baselines
The naive reference point is the official SWE-bench test outcome: a patch counts as correct if it passes the repository's fail-to-pass and pass-to-pass tests. Every false pass is a miss by this baseline by definition. It is the evaluation the field actually uses, and its blind spot is what we are trying to measure.
The strongest existing method for finding hidden wrong patches is UTBoost. It localizes from file to function to line with GPT-4o, generates additional unit tests, and compares candidate and gold behavior on them. On Lite and Verified it found 345 false passes, affecting 40.9% and 24.4% of leaderboard entries, at roughly $1.60 per task. We will run UTBoost's released pipeline on the same set of agent patches, and give our detector the same per-task model-call and dollar budget. Beating it means one of two things. Either our detector recovers a comparable share of UTBoost's confirmed false passes without access to the gold patch, or, combined with UTBoost, it finds confirmed wrong patches that UTBoost alone misses. Confirmation is by manual inspection of a sample plus the hidden tests from Ahmed et al.
For the predictive claim, the strongest simple competitor is a complexity predictor. It is a logistic model on patch features: lines changed, files touched, and a Python adaptation of cyclomatic change complexity over changed basic blocks, motivated by CoREBench's 20.1 versus 9.9 gap. It is trained and evaluated with the same cross-validation folds as a predictor that adds the ladder scores. Beating it means a statistically reliable gain in AUROC for predicting false pass or overfitting once the ladder scores are added to the complexity features, not merely a ladder that works on its own.
The judged-trace competitor is an AgentEval-style evaluation. It applies AgentEval's three-level taxonomy with a GPT-4o judge to the same agent trajectories, and uses the predicted root-cause category as a feature. AgentEval reports failure-detection recall of 0.89 against 0.41 for end-to-end evaluation, measured on its own traces rather than on hidden-regression labels. We will compare recall on our label of false passes, with an equal number of judge calls.
Two ablations isolate the key ingredient. In the first, the rung-two ground truth is restricted to one-hop pairs, mirroring SynFix's synchrony boundary. In the second, rung two is removed altogether. If the predictive gain disappears without multi-hop pairs, the claim that non-local interactions matter is supported. If the gain survives the one-hop restriction, SynFix's coarser graph is enough and we will say so. For the intervention, we compare SynFix-style one-hop neighbor prompting against prompting with the traced pairs, on the same base model. SynFix's reported Verified score of 55.8% with GPT-4o is the reference.
Why not just…?
Why not just generate more tests, as UTBoost does? This is the most serious threat. If augmented testing catches nearly every false pass, the ladder adds diagnosis but no detection. Three points answer it. UTBoost needs the gold patch, which does not exist when an agent edits real code. It reports that patches diverge, not where or why, so it cannot tell a developer which unedited call site broke. And it is not aimed at unmodified reads, where Ojdanic et al. show most change-affected behavior lives. The overlap experiment below settles the matter. If our oracle-free detector finds almost no confirmed wrong patches beyond UTBoost's and recovers few of them without the gold patch, the detector claim fails and the contribution shrinks to the diagnostic benchmark.
Why not just use patch complexity as the predictor of hidden breakage? CoREBench's complexity gap makes this plausible, and we do not argue it away. We include it as a baseline that the ladder must beat incrementally. Complexity measures the task; the ladder measures the model's grasp of the task's interactions. The test is whether two models given the same high-complexity task differ in hidden-regression rate in a way their rung-two scores predict. If adding ladder scores does not improve AUROC over complexity features, our central claim is false.
Why not just ask an LLM judge to review the patch or the trajectory? Judge-based step evaluation works on agent workflows, as AgentEval shows. But a judge reading a diff faces exactly the problem that change interaction errors create: the defect is not visible in the diff. Our rungs are graded by execution, which removes judge calibration as a source of noise. The comparison at matched call budget shows whether a judge reading the trajectory recovers the same signal.
Why not just give the agent more context, such as SynFix's neighbors? If one-hop structure is enough, the multi-hop tracer is unnecessary machinery. The one-hop ablation and the one-hop-versus-traced-pairs prompting comparison test this directly, on the same model and data.
Proposed Experiments
The first experiment is a mechanism census. On the 345 UTBoost false passes and the overfitted patches from Ahmed et al. on SWE-bench Verified, we will run the tracer and measure what fraction of divergences occur at an altered write-read pair in unmodified code, broken down by hop distance. No such number is reported in the file; Gap 3 is exactly that this mechanism has not been checked. If fewer than roughly a third of false passes involve such pairs, change interactions are not the main story, and the rest of the program should be narrowed.
The second experiment builds the ladder over SWE-bench Verified's 500 tasks. We evaluate several agents and base models, report rung-wise accuracy and the ungrounded-correctness rate, and fit the predictive models against the complexity and judge baselines with matched folds. The decisive check is the incremental AUROC of rung two over complexity features for predicting false passes and overfitting.
The third experiment runs the oracle-free detector against UTBoost at equal cost. We report recovery of UTBoost's confirmed false passes and newly found confirmed wrong patches. Results are replicated on SWE-bench Lite and spot-checked on SWE-Gym tasks to guard against benchmark-specific artifacts.
The fourth experiment is the intervention. We feed traced pairs from the candidate patch back to the agent and measure the change in hidden-regression rate and resolution rate, against SynFix-style one-hop prompting on the same model.
The outcome that would most convincingly support the proposal combines three results: a majority of false passes localize to multi-hop unmodified reads, rung-two scores predict them beyond complexity, and the detector finds wrong patches UTBoost cannot without a gold patch. The outcome that would sink it is any of those three failing. Each failure is stated in advance and would be reported as such.
References
- Regression tests to expose change interaction errors
- Mitigating Security Risks in Linux with KLAUS: A Method for Evaluating Patch Correctness
- On the use of commit-relevant mutants
- CoREBench: studying complexity of regression errors
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models
- SynFix: Dependency-Aware Program Repair via RelationGraph Analysis
- UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench
- Investigating Test Overfitting on SWE-bench