Coding agents that repair their own code in a loop, running it, reading the errors and patching, now pass most single-task benchmarks. Over many edits, though, they quietly damage a codebase. Two recent measurement studies show this most clearly. One finds that agents iteratively extending their own code make it more convoluted and more repetitive in roughly three quarters of trajectories. The other finds that the strongest predictor of a later agent failing on top of earlier agent code is input/error contract drift: the agent silently changes what a function accepts, returns, or raises, for example returning None where the original raised an exception. We read this drift as a kind of hallucination. The model invents an intent the user never stated and writes code that is consistent with that invented intent. Unit tests do not catch it, because tests check typical behavior and not the unstated contract.
Our proposal is to pull the contract out as its own reviewable object and spend human attention only there. A separate agent, which never sees the coder's reasoning, detects every change an edit makes to a function's observable contract. It does this mechanically, by running the old and new versions on the same probe inputs. Each change is shown to the developer as a one-line "contract card," such as "parse() on empty input: previously raised ValueError, now returns None." The developer approves or rejects the change, and never reads the code diff unless they choose to. The core bet is that finding drift can be automated, but deciding whether a given drift was intended cannot, because the intent lives only in the human's head. A small, legible question asked at that one point should therefore buy more maintainability per human minute than reviewing whole diffs, and more than any fully autonomous fix.
Core Claim
Contract drift splits into two subproblems that prior systems handle together. Detecting a behavioral change is a program-analysis problem that execution answers deterministically. Judging whether the change was wanted is an intent problem, and that is exactly where language models hallucinate. We claim that a multi-agent loop should automate the first and route only the second to a human, at the granularity of individual contract changes. If this is right, it should lower downstream failure on agent-built code and slow structural erosion. It should also keep the functional pass rate of a strong autonomous loop, and it should do this at a fraction of the human time that full diff review costs. The claim is falsified if either of two automated substitutes, an LLM judging intent or a rule that reverts every unrequested contract change, matches the human-gated version.
Grounding in Prior Work
Two measurement papers motivate the work. SlopCodeBench has agents update their own code across evolving specification checkpoints. It reports that even the best agents pass about 14.8% of checkpoints, and that erosion rises in 77% of trajectories while verbosity rises in 75.5%. Is Agent Code Less Maintainable Than Human Code? builds CodeThread, a set of two-step pull-request chains. It shows downstream resolve rates fall by up to 13.1% when an agent builds on agent code rather than human code, and that contract drift raises failure odds by 1.83x while classical complexity metrics explain little. Neither paper tries an intervention involving humans or role decomposition, and we provide one. Code Hallucination documents related failures in which models write plausible code that ignores atypical constraints or wrongly declares flawed code correct. Those failures motivate our refusal to let an LLM be the final judge of intent.
Three multi-agent systems supply the method we reuse, and their limits constrain the design. AgentCoder separates a programmer from a test designer who never sees the code, which avoids confirmation bias, and it reaches 96.3% pass@1 on HumanEval. Our contract-checking agent borrows that blindness, but it targets contract differences between versions rather than requirement tests. MAGIS splits repository issue resolution into custodian, manager, developer and quality-assurance roles, and it reaches 25.33% on SWE-bench Lite. Its pipeline is fully autonomous and never measures drift. Adversarial Review freezes code while a reviewer and a critic argue with line-cited evidence, and it reaches 75.2% on SWE-bench Verified. The same paper reports that intense review causes over-editing that breaks working contracts. That negative result constrains us: adding more autonomous reviewers looks like part of the cause of drift, not a cure for it.
From the human-AI agency literature, Beyond the Chat (InkSync) shows that small executable edits with warnings cut users' acceptance of inaccurate factual edits by 44% in prose. Earlier work on suggestion granularity (Comparing Sentence-Level Suggestions to Message-Level Suggestions) shows that coarse suggestions make users anchor on them and edit passively. We carry the "small, flagged unit of review" principle from text into code. The unit we choose is semantic, a contract change, rather than a textual diff hunk. AgentClick is the closest system. It intercepts terminal-agent actions as approve/edit cards. However, it shows whole artifacts such as plans and diffs, it has no drift detector, and it reports only qualitative walkthroughs. We differ in what we show the human, in how we choose when to interrupt, and in measuring the effect on maintainability benchmarks.
Research Program
The system has three roles around an existing execution-feedback loop.
The coder is any strong autonomous repair loop. We use an AgentCoder-style programmer and test executor on function-level tasks and an Adversarial-Review-style loop on repository tasks.
The contract steward is the new agent. After each candidate edit, it lists the public and changed functions. It generates probe inputs that cover empty, malformed, boundary and type-violating cases, and runs the old and new versions on them side by side in a sandbox. It records differences in return values, return types, raised exception types, and argument validation. It then uses a model only to phrase each difference as a one-line card and to cluster duplicates. The model never decides whether a difference matters. Changes that the current specification text explicitly requests, matched against the issue or checkpoint description, are labeled "requested." All other changes are "unrequested."
The gate sends unrequested cards to an adjudicator. When a card is rejected, the coder must restore the original behavior while keeping its functional fix, and the steward checks that the restoration worked. Accepted cards become part of a persistent contract record. Later checkpoints are checked against that record, which gives the rollback-to-stable-checkpoint idea from execution-feedback work a semantic rather than textual anchor.
The adjudicator is the experimental variable. It can be a real developer, a simulated human, an LLM, or a fixed rule. For scalable experiments, the simulated human answers each card from ground truth. On CodeThread, a change is "intended" if the human reference code shows the same contract. On SlopCodeBench, it is intended if a later checkpoint's specification or reference solution requires it. This oracle sets an upper bound. A smaller study with 24–32 developers then measures real accuracy, time per card, and sense of ownership, compared against reviewing full diffs.
We will also analyze the cost of the gate. We will count cards per task, and we will vary the probe budget to see how detection recall trades against false alarms.
Baselines
The naive reference point is the coder loop alone, with no steward and no gate. This is the autonomous setting that SlopCodeBench and CodeThread measure. It tells us how much drift and erosion we start from.
The strongest autonomous competitor is Adversarial Review, which reports 75.2% resolve on SWE-bench Verified against 71.6% zero-shot and 72.6% for a five-agent debate. It is the strongest because it is the best-reported method for catching errors through structured multi-agent scrutiny, and it already cites line-level evidence. We will run it with the same backbone model, the same tool access, and a matched token budget, which includes the steward's probing cost. Beating it means two things together: CodeThread contract drift and downstream resolve loss are lower, and SWE-bench Verified resolve is within two points. No reported result was found on Lacuna for Adversarial Review on SlopCodeBench or CodeThread, so we will run it ourselves. For function-level sanity checks, AgentCoder's 96.3% HumanEval pass@1 is the reference.
The strongest human-in-the-loop competitor is full-diff review in the style of AgentClick. The developer approves or edits every code diff the agent proposes. It is run with the same tasks and the same total human minutes as the contract-card condition. Beating it means equal or lower drift at the same time budget, or equal drift in at least 40% less time.
Two ablations isolate the key ingredients. The first replaces the human with an LLM adjudicator that sees the card, the specification and the diff, which tests whether a human is needed at all. The second keeps the human but removes differential execution, so cards are generated by a model reading the diff. This tests whether mechanical detection matters. We also include SlopCodeBench-style quality-aware prompting as a cheap baseline, and a rule that reverts every unrequested contract change.
Why not just…?
Why not just let an LLM judge whether each contract change was intended? This is the main threat, because it removes the human entirely. Our reason for doubt is that intent is under-specified by construction. The checkpoint and issue text is all the model has, and Code Hallucination shows models fill such gaps with memorized canonical behavior. Still, InkSync reports models identifying added external information with 97.5% accuracy, so models can be good at related judgments. The LLM-adjudicator ablation settles the question directly. If its accept/reject agreement with the simulated-human oracle and its downstream CodeThread resolve rates come within noise of the human-gated condition, we will drop the human and report contract-card self-adjudication as the result.
Why not just revert every contract change the specification did not request? This rule is cheap and conservative, and it may well remove most harmful drift. It fails when specifications evolve. SlopCodeBench checkpoints change behavioral contracts on purpose, and a sensible coder may need a change that the current text only implies. We expect blanket reversion to cut drift but lower checkpoint pass rates. If it does neither, it wins and we report that.
Why not just prompt the agent to preserve contracts and write clean code? SlopCodeBench reports that quality-aware prompting lowers initial erosion and verbosity. It does not stop degradation across checkpoints, and it slightly lowers functional correctness. CodeThread's drift finding was obtained with standard agents. We still include a contract-preserving prompt as a baseline, because if prompting alone closes the CodeThread gap, the whole mechanism is unnecessary.
Proposed Experiments
The first experiment runs on CodeThread (1,377 two-step chains). We apply each condition at step one and measure step-two resolve rate, contract drift counts, and the gap between agent-built and human-built code, which reaches up to 13.1% in the original paper. The decisive check is whether the human-gated condition shrinks that gap more than Adversarial Review and the revert rule do, at equal tokens.
The second experiment runs on SlopCodeBench (36 problems, 196 checkpoints). We measure checkpoint pass rate, which is about 14.8% for the best agents, along with the erosion and verbosity slopes across checkpoints. The decisive check is whether the persistent contract record flattens the erosion slope without lowering pass rate, which is the trade-off that quality prompting failed to escape.
The third experiment runs on SWE-bench Verified to confirm that gating costs no more than two points of resolve rate against 75.2%. We will also report cards per issue and an estimate of human minutes.
The user study compares contract cards against full-diff review at matched time. It measures how many seeded drifts are caught, how many intended changes are wrongly rejected, time per decision, and ownership ratings modeled on CoAuthor-style questionnaires.
The overall result counts as positive only if three conditions hold together. The oracle-human condition must beat the LLM adjudicator on downstream resolve. Real developers must reach a meaningful fraction of the oracle's gain. And cards must beat diffs per human minute. If any one of these fails, we will report the failure as the finding.
References
- SlopCodeBench: Measuring Code Erosion Under Iterative Specification Refinement
- Is Agent Code Less Maintainable Than Human Code?
- Code Hallucination
- AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- Beyond the Chat: Executable and Verifiable Text-Editing with LLMs
- Comparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication
- AgentClick: A Skill-Based Human-in-the-Loop Review Layer for Terminal AI Agents