When a panel of language-model agents deliberates over several rounds, some agents change their answers. We cannot tell whether an agent changed its mind because a peer gave a good argument or because it saw that most peers disagreed with it. Human Delphi studies cannot answer this either: you cannot show the same expert, at the same moment, both the panel's median score and a version of the feedback with the median removed. Language-model agents can be shown both, if the deliberation runtime treats each agent's mid-deliberation state as a saved object that can be resumed more than once. We propose such a runtime. It is a Delphi coordinator that snapshots every agent before each feedback step and resumes each snapshot under several counterfactual feedback conditions. It then keeps only the answer changes that are driven by arguments, and rewinds the changes that appear only when the agent sees the majority.
The opportunity is concrete. Voting or Consensus? Decision-Making in Multi-Agent Debate found that forcing consensus on multistep reasoning collapses MuSR accuracy to 28.4% against 54.8% for voting, because a lone correct reasoner gets talked out of its answer. Debate or Vote proves that unguided debate is a martingale: an agent's expected belief in the right answer does not improve across rounds. On a Qwen2.5-7B suite, zero-round majority voting (0.7691) beat two-round debate (0.7377). Both papers point to the same missing capability. A deliberation protocol needs a way to tell which updates carry information and which only spread social pressure, and a way to undo the second kind.
Core Claim
Delphi feedback has two parts: summary statistics (the median and spread of the panel's answers) and qualitative rationales. The two parts are separable, and the separation can be measured per agent and per round by resuming the same saved agent state under three conditions. The first is full feedback. The second is rationales only, with the statistics and vote counts removed and the rationales shuffled. The third is a null condition that re-asks the question with no feedback, which measures plain resampling noise. Our claim is that answer changes appearing under full feedback but not under rationales-only feedback are mostly harmful, meaning right-to-wrong flips. Our second claim is that a protocol which reverts those changes, and accepts only changes that beat the null condition, keeps correct minority answers alive. It should then outperform compute-matched majority voting on reasoning tasks where Kaesberg et al. found consensus to be destructive.
The decision rule fits in one line. Let $p^{F}_{i,r}(y)$, $p^{R}_{i,r}(y)$, and $p^{\varnothing}_{i,r}(y)$ be the fractions of $K$ resumptions of agent $i$'s round-$r$ snapshot that switch to answer $y$ under full feedback, rationales-only feedback, and no feedback. The coordinator accepts a switch to $y$ only if
$$p^{R}_{i,r}(y) - p^{\varnothing}_{i,r}(y) > \tau,$$
where $\tau$ is a threshold tuned on held-out data. Otherwise it restores the agent to its snapshot. The gap $p^{F}_{i,r}(y) - p^{R}_{i,r}(y)$ is reported as that update's conformity score and is never used to accept a change.
Why this needs resumable state
This idea comes from the closure and continuation literature, where a continuation is a saved "rest of the computation" that can be resumed later. The useful lessons are about structure. Bobrow's tree of frames and MAGMA-LISP showed that separating variable bindings from control flow lets many branches fork from one parent without corrupting it. Effect handlers make the structure explicit. In Continuing WebAssembly with Effect Handlers, a computation performs an operation such as suspend, and a surrounding handler decides how and when to resume it. We model one agent's Delphi round as a computation that performs a consult effect when it needs peer feedback. The coordinator is the handler, and the counterfactual design is multi-shot resumption, meaning the same suspended agent is resumed three times with different feedback.
WasmFX restricts itself to single-shot continuations because multi-shot ones break linear memory safety. For a language model, that restriction disappears. The agent state is an immutable token prefix, so multi-shot resumption is safe by construction. The real cost is recomputing the model's cached attention state, which is the replay cost that Capturing the future by replaying the past identifies as the weakness of replay-based continuations on deep branches. Our runtime therefore stores agent states as a persistent prefix tree whose branches share cached attention state, the analog of the tree of frames. The coordinator keeps a worklist of suspended agents waiting on feedback, in the way Tabling as a library with delimited control suspends dependent goals with shift/1 until new answers arrive. These papers supply method. None of them touches deliberation.
The scientific claim does not depend on the programming-language framing. The framing makes the counterfactual protocol cheap and correct, and we measure that contribution separately.
Relation to prior work
Tree of Thoughts searches over a single agent's intermediate reasoning steps with backtracking. We do not search over thoughts. We branch over feedback conditions across a panel, and we backtrack based on why an agent's answer changed, not on how good a state looks. Kaesberg et al.'s All-Agents Drafting (independent first answers, +3.3%) is the Delphi blind first round. We adopt it unchanged, but their later rounds are linear and append-only and cannot be undone. MAD-Conformist in Debate or Vote breaks the martingale by anchoring agents toward the majority. We break it in the opposite direction, by suppressing majority-driven movement. Comparing the two directly is informative: their gain is from 0.7332 to 0.7625.
Consensus Maximization Tree Search Revisited motivates group-level pruning, but its geometric setting does not transfer, and we do not use it. The clinical Delphi literature defines the protocol we emulate. Examples are the median-and-interquartile-range feedback in An international, Delphi consensus study to identify priorities for methodological research in behavioral trials and the conformity concern raised in panels such as A Delphi consensus statement for digital surgery. Our results bear on human Delphi design only indirectly.
Research Program
We will build the runtime on an open-weight serving stack with prefix caching. It has three parts. Agent states are nodes in a persistent prefix tree. The consult effect suspends an agent. A coordinator handler resumes each suspended agent $K$ times under each of the three feedback conditions. Rationales for the rationales-only condition are anonymized, shuffled, and stripped of counts and of phrases like "most agents."
The protocol runs a blind first round, then up to three feedback rounds. Delphi's stability rule terminates early: stop when no agent's accepted answer changes in a round. Final aggregation is a plurality vote over accepted states.
The analysis has three parts. First, we decompose every answer change into wrong-to-right and right-to-wrong flips, separately for the full-feedback and rationales-only conditions. Second, we test whether conformity score predicts right-to-wrong flips. Third, we track how often a correct answer held by a minority in round one survives to the final answer.
Models are Llama 3 8B and 70B, to match Kaesberg et al., and Qwen2.5-7B, to match the martingale study.
Baselines
The naive reference is a single agent answering once with chain-of-thought. It sets the floor and calibrates how much any multi-agent method adds.
The strongest competitor is compute-matched majority voting over independent samples. This is zero-round voting from Debate or Vote, which reports 0.7691 average accuracy on Qwen2.5-7B, and the same idea as Self-Consistency, which reports 74.4% on GSM8K with PaLM-540B. Our protocol issues roughly $3K$ resumptions per agent per round, so majority voting receives the same total number of generated tokens as extra samples. We measure prefill and decode tokens, not just call counts. Beating it means a higher accuracy at equal token budget, on at least MuSR and the reasoning half of the suite, with confidence intervals from bootstrap over items.
The protocol baselines are Kaesberg et al.'s voting and consensus protocols with All-Agents Drafting. On MuSR they report voting at 54.8% and consensus at 28.4%, averaged over their Llama 3 settings. We also run MAD-Conformist. All are rerun with our models, prompts, and round limits rather than copied, and their reported numbers serve only as sanity checks.
Two ablations isolate the key ingredient. The first is linear rationales-only Delphi: every round shows only stripped rationales, with no forking and no gating. This tests whether removing statistics alone does the work. The second is fork-without-null: we accept changes that survive the rationales-only condition, but drop the no-feedback control. This tests whether separating argument-driven change from resampling noise matters. For the systems claim, the baseline is replay, which rebuilds each branch by re-prefilling the full transcript.
Why not just…?
Why not just take a majority vote over more independent samples? This is the real threat. On short-answer tasks it may win, since the martingale result says debate adds noise there. Our bet is narrower. On MuSR-style multistep reasoning, correct answers are often minority answers in round one, and argument exchange can spread them while voting cannot. The compute-matched comparison settles this directly. If we lose on MuSR at equal tokens, the method has no use case and we will say so.
Why not just strip the statistics from the feedback and run ordinary Delphi? This is cheaper and may capture most of the gain. The linear rationales-only ablation tests it. Forking adds two things stripping cannot. The null condition rejects changes that are only resampling noise, which the martingale analysis predicts are common. The conformity score also gives a per-update diagnostic. If the ablation matches the full method on accuracy, we will report the forking machinery as an analysis tool rather than a performance method.
Why not just save the transcript and re-prompt, without any continuation machinery? For language models a transcript prefix is a perfectly good continuation, so semantically this is the same protocol. The difference is cost: replay re-prefills shared history on every branch, which grows with rounds and panel size. We measure prefilled tokens and wall-clock time for the shared-prefix tree against replay. The systems contribution stands or falls on that number alone.
Why not just use Tree of Thoughts with several agents as evaluators? Tree of Thoughts branches on reasoning content and prunes by self-evaluation. It has no notion of why an agent changed its answer, so it cannot distinguish persuasion from pressure. Where its tasks fit, such as Game of 24 (74% with GPT-4), our method is orthogonal, and we do not claim to beat it there.
Proposed Experiments
The benchmarks are MuSR, with all three domains reported separately. Human accuracy there is about 95% and GPT-4 reaches 80.4% on murder mysteries, per MuSR. We also use the Kaesberg suite (MMLU, MMLU-Pro, GPQA, StrategyQA, SQuAD 2.0) and the seven-benchmark suite from Debate or Vote, including GSM8K and CommonsenseQA. As a secondary, Delphi-shaped setting, we use the Contextual Integrity Synthetic Dataset (729 norm examples). There, agents give Likert ratings of how appropriate an information flow is, and we measure whether minority variance survives without loss of agreement with labels. No reported result was found on Lacuna for multi-agent or Delphi methods on this dataset.
The metrics are accuracy, flip decomposition by condition, correct-minority survival rate, tokens per item, and rounds to stability.
The decisive check comes first and is cheap. Before any accuracy comparison, we run the three-condition fork on MuSR round-two updates and ask whether right-to-wrong flips concentrate in updates with high conformity score. If conformity score does not separate harmful from helpful flips, the core claim is false, and gating cannot help. If it does, the second check is whether the gated protocol beats compute-matched majority voting on MuSR and closes part of the 26-point gap between consensus and voting reported by Kaesberg et al. The third check is whether the shared-prefix runtime cuts prefilled tokens substantially relative to replay as rounds increase.
References
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?
- MAGMA-LISP: A "Machine Language" For Artificial Intelligence
- Continuing WebAssembly with Effect Handlers
- Capturing the future by replaying the past (functional pearl)
- Tabling as a library with delimited control
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- Consensus Maximization Tree Search Revisited
- An international, Delphi consensus study to identify priorities for methodological research in behavioral trials in health research
- A Delphi consensus statement for digital surgery
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning