When Passing Checks Do Not Establish Completion: Evidence Interpretation in Coding Agents

Xueping GaoarXiv 2026 · arXiv

TL;DR

Passing a test suite doesn't mean an AI coding agent has actually fixed the bug. This paper demonstrates that agents often produce "plausible" patches that satisfy test assertions through unintended side effects or misinterpreted execution facts, highlighting a significant gap between a passing "PASS" verdict and true task completion.

What Problem Does This Paper Address?

In the world of Automated Program Repair (APR), there is a long-standing distinction between plausible patches (those that pass the tests) and correct patches (those that actually fix the underlying logic). As we move toward autonomous coding agents, we rely heavily on their ability to interpret test results to claim a task is "done."

The authors tackle the "Oracle" problem: Does a passing check actually establish completion? They argue that verification procedures in coding agents are often shallow. An agent might claim success because a test passed, but that "PASS" might be a false positive caused by:

  1. Startup Obstruction: The edit broke the test runner's initialization, leading to an "UNKNOWN" state that the agent misinterprets.
  2. Witness Misattribution: The evidence of success (the "witness") came from a side effect (like a file cleanup) rather than the intended operation.
  3. Fact Errors: The agent correctly predicts the test outcome but hallucinates the execution facts that led to it.

Methods

The researchers employed a rigorous "forensic" approach to analyze 118 repair episodes across 31 repository tasks. Their primary tool was source intervention:

  • Component Reversions: Systematically undoing parts of an agent's patch to see which specific lines actually caused the test to pass.
  • Startup-Reference Substitution: Replacing the agent's modified startup code with original "known-good" code to see if the "fix" was actually just bypassing a broken initialization path.
  • Operation-Isolating Consumers: Creating custom, "narrow" tests that check for the intended behavior (e.g., "did an explicit flush happen?") rather than just the broad assertion the original test used.
  • Lifetime and Resource Controls: Manipulating environment variables like system clocks (mtime) or object destruction sequences to see if the agent's "fix" relied on fragile, unintended timing or cleanup behaviors.

![Fig.

  1. Production code participates in both the behavior under test and the production of validation evidence.](/figures/art_1f3050bf0b0d400fa364d071cd537f9d)

As shown in Figure 1, the authors emphasize that production code isn't just the subject of the test; it's a participant in the validation process itself.

Datasets

The study utilized several cohorts of tasks (summarized in Figure 3/Table I) primarily drawn from real-world Python and Rust repositories (e.g., Django, Pylint, pytest, zstd).

  • Models Evaluated: Qwen3-Coder-30B and Devstral Small 2 24B.
  • Calibration: They also analyzed public submissions from the SWE-agent benchmark to ensure their findings weren't limited to their own experimental setups.

Key Insights and Ideas

  • The "Flush" Fallacy: In one Django task (Django13516), the agent (Qwen) added a flush command to a write() method. The test passed! However, interventions showed the test passed because of a side effect during object cleanup, not because the new code worked. In fact, the agent's "fix" actually broke compatibility with other output types.
  • Facts $\neq$ Verdicts: An agent can be right about the result (the test passed) but wrong about the reason. Figure 2 illustrates this: out of 122 correct "PASS" labels, 16 contained factual errors about what the code actually did, and 11 claimed success for things outside the authorized scope.

![Fig.

  1. Receipt-supply output partition showing fact errors and scope excess.](/figures/art_35b5270296a045a29a8aff6e6e152c00)
  2. Reachability as a "Fix": Sometimes an agent's edit makes a test pass simply because it prevents the test from ever reaching the buggy code. By breaking the "startup" path, the agent creates a false sense of success.
  3. Explanations are Unreliable: Even when an agent successfully repairs a bug, its generated "explanation" or "notes" often contain false claims. Repair success does not certify the agent's diagnostic reasoning.

Key Results

  • Factual Accuracy: In a prospective study, agents misreported factual fields in 20% of responses, even when they got the final verdict (PASS/FAIL) right.
  • Witness Attribution: In the "flush" repair case (Table III/Figure 5), the Qwen model's patch passed the official check but failed when a more rigorous "explicit behavior" check was applied. It produced 0 explicit flush events while the "Official fixed" code produced 1.
Table III. Flush evidence and behavior on one issue.
  • No "Skill" Advantage: The authors tested whether providing agents with "Skills" (pre-packaged procedures) or generated notes improved performance. They found no consistent overall advantage, suggesting that better "reasoning" doesn't necessarily fix the underlying interpretation issues.

Limitations and Future Work

  • Scope: The study is limited to 31 tasks and two model families. While deep, it doesn't claim to estimate the prevalence of these issues across all of GitHub.
  • Post-hoc Analysis: Many of the "consumer" tests used to catch agent errors were designed after seeing the agent's failures.
  • Future Directions: The authors suggest that agents need "evidence-carrying" representations—code that doesn't just pass a test but proves it satisfied a specific contract. They advocate for moving beyond simple PASS/FAIL metrics toward more granular verification of execution stages and resource lifetimes.

Drafting assistance for the original paper was provided by OpenAI Codex [50].

Summary updated

Abstract

Verification procedures in coding agents must preserve execution facts, interpret their scope and discharge checking obligations. We study these responsibilities through qualified repository checks, controlled evidence consumers and executable workflows. A prospective thirteen-task factorial study finds that correct reported facts need not yield the requested finite verdict. Conversely, all 122 delivered labels in a held-out receipt-supply study are correct, while sixteen responses misreport facts and eleven others exceed authorized scope. Production repair supplies successful controls and finite-supported preservation failures. Across 118 repair episodes on 31 tasks, source interventions separate startup reachability from repair and witness production from the claimed operation. Lifetime and resource controls delimit what passing validation establishes. Generated-note comparisons yield local repair differences but can state false explanations; strong ordinary baselines, new tasks and Skill-transfer studies supply no consistent overall advantage. These results characterize evidence interpretation, applicability and fulfilled actions while limiting claims of general diagnostic superiority, Skill deterioration or deployed folding effects.
BibTeX
@misc{gao2026when,
  author = {Xueping Gao},
  title = {{When Passing Checks Do Not Establish Completion: Evidence Interpretation in Coding Agents}},
  year = {2026},
  eprint = {2611.22835},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22835},
}
Figure