Coder Disagreement as a Probe for Underspecified Plans

When a team of language-model agents writes code, the usual split is that a planning agent turns the task description into step-by-step instructions and a coding agent turns those instructions into a program. The most common way this breaks is not a bad coder. The plan leaves something out, and the coder quietly fills the hole with a plausible guess. This is a hallucination in the strict sense: confident content that nothing in the input supports. It also produces the "silent failure" typical of generated code, a program that runs cleanly and computes the wrong function. We propose a cheap, execution-grounded test for these holes. Hand the same plan to several independent coders, run their programs on automatically generated inputs, and find the inputs where they disagree. Any such input shows that the plan admits more than one behavior. We send that evidence back to the planner, not the coder, and ask it to settle the ambiguity in the plan.

The key design choice is that no language model is ever asked to predict a correct output. The system only needs inputs, and an LLM can propose inputs without hallucinating answers. This avoids the "feedback trap" that pushed earlier systems either to trust unverified LLM-written tests or to restrict themselves to the few examples in the prompt. The claim is simple enough to be proven wrong. Disagreement among coders given a fixed plan should predict eventual failure. Repairing the plan at the disagreeing inputs should recover more failures, at matched compute, than voting among many samples or adding an LLM reviewer.

Core Claim

Disagreement among coders measures how underspecified the plan they share is, and that measurement points at the right place to intervene. Two strands of evidence in the file motivate this. The first is Testing and Enhancing Multi-Agent Systems for Robust Code Generation, which fuzzes multi-agent coders with meaning-preserving prompt rewrites. It finds drops of 7.9% to 83.3% and traces 75.3% of failures to the "planner-coder gap," meaning abstract plans that coders misread. The second is the "trap of partial anchoring" reported by Tracing the Cascade: fragmentary context makes a model more confident without constraining it enough, which raises hallucination severity. A terse plan is exactly such a partial anchor for the coder. That same paper finds that early errors have low repairability downstream, which argues for fixing the plan rather than patching code. The probe also rests on work showing that models become inconsistent when cross-examined about content they hallucinated (Agrawal et al.). We apply that idea to programs: independent coders filling the same hole tend to fill it differently, and execution makes the difference observable.

The claim has a known blind spot, and we measure it rather than hide it. If every coder makes the same wrong guess, the probe sees nothing. Code Hallucination documents such a shared guess in the "memorization trap," where models solve the familiar LeetCode version of a problem and ignore the modified constraint. We report how many failures fall into this agreed-but-wrong class, because it bounds what the method can ever achieve.

Research Program

The system has four parts. A planner writes a numbered plan from the task description. $K$ coders, drawn from different models or sampling temperatures, each implement the plan independently. They are asked to tag code regions with the plan step they implement and to log intermediate values at step boundaries. An input generator, which sees only the task description and never any code, proposes a few hundred inputs by random, boundary, and structural generation. It never supplies expected outputs. A deterministic harness runs every program on every input, groups programs by behavior (identical outputs or identical exception class), and extracts a small set of distinguishing inputs.

When more than one behavior group exists, the planner gets the distinguishing inputs, the competing outputs, and the plan step where the logged intermediate values first diverge. It is asked to revise that step so that exactly one behavior follows from the task description. The loop repeats until the coders agree or the budget runs out. The final program comes from the largest group that also passes the task's sample input-output pairs.

The step-level localization reuses the idea behind AgentHallu, which benchmarks finding the trajectory step where a hallucination began and shows that LLMs manage this poorly: 41.1% at best, and 11.6% on tool-use steps. We replace LLM judgment with divergence among logged values, and we report how often our localization matches human labels of the underspecified step on a sample of 200 tasks.

We also study the probe as an instrument, separately from its use in repair. For each plan we record the disagreement rate, defined as the fraction of generated inputs on which the coders split into more than one group. We then ask whether it predicts final failure better than planner self-reported confidence (which MapCoder already elicits) and better than plan length. Using the CodeHalu taxonomy from CodeHalu, we break down which hallucination types (mapping, naming, resource, logic) the probe catches. We expect it to be strongest on logic deviations and requirement omissions and weakest on shared memorized errors.

Baselines

The naive reference is a single model prompted directly, with no plan and no agents. AgentCoder reports zero-shot GPT-4 at 52.2% Pass@1 on MBPP-ET. We rerun this on our backbone rather than quote it.

The strongest published planner-coder pipeline is MapCoder, which uses retrieval, planning, coding, and debugging agents and backtracks across alternative plans when sample tests fail. It reports 28.5% Pass@1 on CodeContests and 93.9% on HumanEval with GPT-4, using about 16.7 API calls per problem. It is the most direct competitor because it already revises plans. The difference is that it revises only when sample tests fail, so it is blind when the samples pass but the plan is ambiguous. AgentCoder is the strongest test-based competitor. It uses a test designer that sees only the description, and it reports 96.3% on HumanEval and 91.8% on MBPP. Its tests, however, carry LLM-predicted expected outputs that can themselves be hallucinated.

The closest method to ours is the Monitor Agent from Testing and Enhancing Multi-Agent Systems. It is an LLM layer placed between planner and coder that interprets the plan and checks code statically, and together with multi-prompt generation it recovers 40.0% to 88.9% of fuzzing-induced failures. Our proposal differs from it in exactly one respect: the check between planner and coder is grounded in execution rather than in another LLM reading. That makes the Monitor the comparison that matters most.

All systems run on the same backbone models, the same task splits, and the same token budget per problem. Budget-matched variants get extra samples or rounds until their token spend equals ours. Each system receives the same prompt-tuning effort, three prompt variants selected on a held-out development split. We count as beating a baseline a Pass@1 gain that holds across three seeds on CodeContests and on the mutated CodeContests split, at equal or lower tokens. A smaller robustness drop under mutation also counts. On HumanEval-class benchmarks, where published scores are already above 93%, we treat gains as secondary.

Three ablations isolate the key ingredient. The first sends the disagreement evidence to a coder instead of the planner, which tests whether the plan is the right place to fix. The second replaces distinguishing inputs with the same number of random generated inputs, which tests whether disagreement carries the information. The third removes step localization, which tests whether pointing at a specific plan step matters.

Why not just…?

Why not just sample many programs and pick the largest behavior group? This is the most serious threat. Behavioral voting uses the same execution harness and the same generated inputs, but it never revises the plan. If holes in the plan are filled at random, voting may already choose the right fill. If the fill is biased toward a plausible wrong answer, voting locks that wrong answer in, and only a planner reading the task again can recover. The budget-matched comparison between voting over $N$ samples and our repair loop settles this directly. We commit to one rule: if voting matches us within seed noise on CodeContests and on mutated CodeContests, we will report the probe as a failure-prediction instrument only and drop the repair claim.

Why not just add an LLM reviewer between planner and coder? That is the Monitor Agent. It is itself a prompt layer that can miss logic bugs, and Code Hallucination shows frontier models failing to spot seeded bugs in merge sort, which is a warning against LLM-only review. We run the Monitor under our budget. We also combine it with our probe, to test whether the two catch different failures or the same ones.

Why not just give a stronger reasoning model more thinking time or more compiler feedback? VERINA finds that 64 rounds of compiler feedback lift Lean proof success only to about 20%, with models "thrashing." CodeHalu finds syntax errors at 0.0020 while logic deviation dominates, so feedback from running and compiling code has little to say about wrong-but-runnable programs. Still, a single model spending our whole budget on long reasoning is a fair competitor, and we include it.

Why not just clarify the specification from its test examples first? µFiX does this in a single-agent pipeline. It derives a specification from the embedded tests and checks it before coding, and it reports large relative HumanEval gains. The approach depends on examples in the prompt, and 81.57% of its benchmark problems have at least three. We run µFiX as a front-end baseline and also stack it before our planner. We further evaluate on a variant of CodeContests with the sample examples removed, where we expect µFiX to degrade and our probe to hold, since the probe needs inputs only.

Proposed Experiments

The main benchmarks are CodeContests, xCodeEval above difficulty 1000 (where MapCoder reports its gains fading), and HumanEval+ and MBPP-ET as saturated sanity checks. For robustness we regenerate the Rephrase, Insert, Expand, and Condense mutations from Testing and Enhancing Multi-Agent Systems on CodeContests and report the Pass@10 drop against the unmutated set. For the hallucination breakdown we run our generations through CodeHaluEval's 699 APPS-derived tasks. There GPT-4 has a reported 33.04% hallucination rate, and we report the change in rate per category.

Two checks decide the fate of the project. The first is whether plan-level disagreement predicts final failure, with an AUROC clearly above planner self-confidence. If it does not, the core claim is false and nothing downstream matters. The second is whether the repair loop beats budget-matched behavioral voting and the Monitor Agent on CodeContests and on its mutated version, and whether the "send evidence to the coder" ablation loses most of that gain. If both checks pass, the result is a specific, reusable finding: in multi-agent code generation, hallucinations are best found by making coders disagree and best fixed where they originate, in the plan. We report the agreed-but-wrong fraction alongside, as the honest ceiling on that finding.