Coding agents now fix real bugs by trial and error. They write a patch, run tests, read the failures, and try again. The trouble is that for a new bug report there are usually no trustworthy tests. The agent writes its own "reproduction" tests and then optimizes against them. Doing this repeatedly makes things worse: patches drift toward passing the agent's flawed tests rather than fixing the bug. Putting a human in the loop seems like the obvious remedy. Yet a large body of human-AI teaming research finds that human-plus-AI teams routinely do worse than the AI alone, because people accept persuasive but wrong AI output, or override correct output out of stubbornness.
We propose a specific division of labor that targets both failures. The agent generates many candidate patches. The system then searches for concrete inputs on which those candidates behave differently. It shows the developer one such input, the bug report, and the two competing outputs, and asks only "which behavior is correct?" Each answer becomes a trusted test that eliminates candidates and steers further refinement. The developer never reads a diff and never sees the agent's explanation or preference. The claim is that this kind of question is cheap and checkable for a human, and that its answers carry information the agent cannot produce itself. If so, a few seconds of human judgment per bug should lift a strong autonomous system above its own ceiling, which a human reviewing patches with the same time budget does not do.
Core Claim
Teaming research gives a sharp condition for when humans help. The meta-analysis in In Search of Verifiability reviewed roughly 30 user studies of AI-advised decisions. It found that every study reaching complementary performance, meaning the team beats both the human and the AI alone, gave humans a way to verify the AI's answer more cheaply than solving the problem from scratch. Studies that gave narrative or feature-importance explanations did not reach it. Does the Whole Exceed its Parts? adds that explanations of the AI's top choice can lower human accuracy when the AI is wrong, acting as a "sales pitch."
Reading a patch diff with an agent's justification is exactly that sales-pitch setting. Judging whether parse("1e-3") should return 0.001 or raise an error, given the issue text, is a verification question. It asks about intended behavior, which is precisely what the agent is unsure about.
The second ingredient is about where human input is worth spending. Human Expertise in Algorithmic Prediction shows that a team provably beats an algorithm when human input is used only on inputs the algorithm cannot tell apart. Inputs where the agent's own candidates disagree are the code analogue of those blind spots: the agent's own signal there is split. Asking the human there and nowhere else is the selective conditioning that the theory supports.
The concrete claim is this. With a budget of three or fewer behavioral questions per issue, human-answered disagreement queries should cut test overfitting on TDD-bench Verified toward the level reached when the agent is given the hidden developer tests. They should also raise resolution above the best autonomous selector run on the same candidate pool. A human reviewing patches with equal time should fail to do either.
Research Program
The work has three parts.
The query engine. For each issue, an agent produces a pool of candidate patches. The pool is sized to match the 16-sample regime of SWE-Gym. The agent also writes candidate probe calls: small snippets that call the affected functions on varied inputs. We run every probe against every candidate in the benchmark's Docker containers and group the candidates by their observed behavior, which includes return values, raised exceptions, and printed output.
A probe is informative when it splits the surviving candidates into large behavior groups. We pick the probe that maximizes the expected number of candidates eliminated. This plays the role of the query-selection committee in LEARN2FIX (Human-In-The-Loop Automatic Program Repair). The difference is that LEARN2FIX learned a numeric constraint oracle for small C programs. Here the "committee" is the set of candidate patches, which makes the method work on arbitrary Python objects and multi-file repositories.
The interface. The developer sees the issue text, the probe call, and two or three distinct outputs in random order. They answer "A", "B", "neither", or "can't tell". The interface deliberately shows no diffs, no agent rationale, and no confidence scores. An ablation adds back the agent's preferred answer to measure anchoring.
Each answer becomes an assertion. Candidates that violate it are discarded. If no candidate survives, the assertion joins the agent's test set and the agent runs another refinement round.
The analysis. We measure resolution against the hidden developer tests, the overfitting rate as defined in Is the Cure Still Worse Than the Disease?, human time per issue, and human answer accuracy. We also measure the error correlation between human answers and an LLM answering the same queries, following Robust Human-AI Complementarity under Uncertainty. That paper found a correlation of r = 0.963 between human and LLM errors on ForecastBench and showed that prompting the model to be contrarian did not lower it. The open question is whether questions grounded in execution, rather than in text, produce lower correlation. If the correlation stays near that level, the human adds little, and we will report that.
Baselines
Agent alone (the do-nothing reference). This is the same agent's iterated self-refinement against its own generated tests, with no human. It is the setting of Is the Cure Still Worse Than the Disease?, which reports overfitting on TDD-bench Verified rising from 21.8% to 25.5% for Claude-3.7 Sonnet over 15 refinement iterations. The same paper reports 5.8% when the agent has access to the hidden golden tests. That figure is our practical ceiling. We rerun this pipeline with our backbone on the same 449 instances rather than comparing against the published numbers directly.
Strongest autonomous selector. We train an outcome reward model to pick the best of the 16 candidates, as in Training Software Engineering Agents and Verifiers with SWE-Gym. That paper reports 32.0% resolution at Best@16 on SWE-bench Verified with Qwen-2.5-Coder-32B, against 20.6% at Pass@1 and a 42.8% Pass@16 ceiling. That ceiling bounds any method that selects from the pool; our refinement step can in principle exceed it. We reuse the identical candidate pool for every condition, so selectors are compared on equal footing. Beating this baseline means a statistically reliable resolution gain over the trained selector on the same pool, with at most three human queries per issue.
We also report Adversarial Review, which reaches 75.2% on SWE-bench Verified. It uses a stronger backbone and roughly 4.5 times more tokens, so we treat it as a reference point and run a token-matched version on our backbone where feasible.
Ablations isolating the key ingredient. The first ablation has an LLM answer the identical queries. The second uses majority vote across behavior groups, so the most common behavior wins with no oracle at all. If either matches the human condition, the human is unnecessary. The third randomizes probe selection, which tests whether targeting disagreements matters.
Human patch review. Developers get the same per-issue time budget and see the top-ranked candidate diffs with agent explanations. They choose a patch or reject all of them. This is the conventional human-in-the-loop design, and the teaming literature predicts it will underperform.
Why not just…?
Why not just scale sampling and a verifier? Best@16 closes only part of the distance between 32.0% and the 42.8% ceiling. More importantly, the reward model judges code plausibility, not intended behavior, so it cannot settle cases where two candidates disagree about what the issue meant. The experiments will show whether this is true. If the selector already picks the correct behavior group on most disagreement queries, our gain will vanish. We report the selector's accuracy restricted to those queries.
Why not just use behavioral consensus without a human? This is the most serious threat. Majority vote over execution behavior is cheap and could recover much of the benefit. Our counter-hypothesis is that agents share systematic misreadings of the issue, so the majority is often wrong on exactly the hard cases. We run majority vote and the LLM answerer on identical queries. If either comes within noise of human answers, we will conclude that the gain comes from disagreement search rather than human input, and say so.
Why not just have the developer review the final patch? That is the setting where the teaming literature finds sales-pitch effects. Comparing Developer and LLM Biases in Code Evaluation also shows that code-quality judgments are noisy and shaped by verbosity. The human patch-review condition, run at matched time, tests this directly.
Why not just ask the developer to write the tests? This is too costly at the time budgets we target. We include a capped human-writes-one-test condition on a subset to price it.
Proposed Experiments
Simulated oracle stage. We first run all conditions on TDD-bench Verified and SWE-bench Verified with a simulated answerer that uses the reference patch's behavior. This bounds the achievable gain at query budgets of 1, 3, and 5, and checks whether disagreement queries exist for most issues. The method is useless where candidates never diverge, and we report that coverage.
Human study. Professional developers then work on a stratified subset of about 100 issues, assigned within subjects across the human-query, human-review, and capped-solo conditions.
Decisive checks. First, overfitting under human queries must fall clearly below the agent-alone rate, approaching the 5.8% golden-test reference. Second, resolution must exceed both the trained selector and the LLM-answered queries on the same pool. Third, the team must beat the human's time-capped solo performance, which establishes true complementarity rather than mere AI improvement. Fourth, human-LLM answer error correlation must be measurably below the forecasting regime. Showing the agent's preferred answer should lower human accuracy on the queries where the agent is wrong. If it does, that confirms the interface design matters, not just the presence of a human.
References
- In search of verifiability: Explanations rarely enable complementary performance in AI-advised decision making
- Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
- Human Expertise in Algorithmic Prediction
- Training Software Engineering Agents and Verifiers with SWE-Gym
- Human-In-The-Loop Automatic Program Repair
- Investigating Test Overfitting on SWE-bench
- Robust Human-AI Complementarity under Uncertainty
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- Comparing Developer and LLM Biases in Code Evaluation