When several language-model agents work on the same hard question, they tend to fail in one of two ways. They may all converge on the loudest answer: one confident wrong agent can drag the whole group with it. Or they may waste their effort, with every agent re-checking the same leading answer while a correct minority view goes unexamined. The Delphi method was designed for the human version of these problems. It is a structured expert-consensus procedure: panelists answer anonymously, a coordinator returns only statistical summaries and anonymous rationales, and the process repeats until the ratings stabilize. Parallel exploration has solved the second problem in a different setting. When many workers each draw an independent random sample from a shared belief and act on it (Thompson sampling), they spread out across promising options without any need to talk to each other.
This note proposes to combine the two. A Delphi coordinator keeps a probability distribution over candidate answers. Whenever any agent becomes free, the coordinator draws one sample from that distribution and assigns the agent to scrutinize the sampled answer, either verifying it or refuting it. The agent's verdict updates the distribution immediately, without waiting for the other agents. The claim is that this panel matches the adversarial robustness of the best synchronous consensus protocol while fixing that protocol's main failure mode, which is discarding correct minority evidence. It should also reach a stopping decision with less wall-clock time and fewer tokens.
Core Claim
The closest existing work is Self-Anchored Consensus (SAC) from Robust Multi-Agent LLMs under Byzantine Faults. In SAC, agents broadcast answers, each receiver scores incoming messages with its own model, drops the $F$ lowest-scoring ones, and refines its answer. This defends against "Byzantine" agents, meaning adversaries that assert false answers with self-reported confidence 1.0. It has two limits. First, it runs in lockstep synchronous rounds. Second, the authors acknowledge an anchor weakness: an agent whose initial belief is wrong will score correct peers low and filter out exactly the evidence it needs.
Our protocol differs in three ways. First, filtering is replaced by allocation. No agent decides whose messages to ignore. Instead, the coordinator decides which hypothesis each agent must examine. Minority answers are therefore guaranteed a verification rate proportional to their current posterior mass. Second, following Delphi's controlled-feedback principle, sender confidence never enters the aggregate. Only verdicts on assigned hypotheses count, and each agent's total influence is capped. Agents are anonymous to one another but accountable to the coordinator. Third, updates are event-driven, in the style of asynchronous Thompson sampling, so slow agents never stall the panel.
Concretely, the coordinator clusters answers into candidate hypotheses $h$. It gives each hypothesis a Beta belief with parameters $(a_h, b_h)$ over the probability that $h$ survives scrutiny. When an agent frees up, the coordinator draws $\theta_h \sim \mathrm{Beta}(a_h, b_h)$ for every $h$ and assigns the agent to
$$h^\star = \arg\max_h \theta_h .$$
A pass verdict increments $a_{h^\star}$ and a fail verdict increments $b_{h^\star}$. Each agent's contribution to any single hypothesis is capped at $c$ units, so $F$ adversaries can move any belief by at most $Fc$ pseudo-counts. The panel stops when the estimated probability that the leading hypothesis is best exceeds a threshold $\tau$ across $k$ consecutive updates. This is the posterior analogue of the Delphi stability criterion, which ends deliberation when ratings stop moving rather than after a fixed number of rounds.
Why These Ingredients
Each ingredient answers a specific piece of prior work.
Motivating evidence for randomized allocation. Deterministic exploration makes parallel workers duplicate each other. This was observed for Upper Confidence Bound (UCB) policies in Parallel Contextual Bandits in Wireless Handover Optimization. It is also why Society of Agents: Regret Bounds of Concurrent Thompson Sampling can prove a $1/\sqrt{n}$ per-agent regret speedup for $n$ concurrent Thompson-sampling agents. Its bound assumes free, instantaneous synchronization, a limit we do not inherit because we measure latency directly.
Methods for non-blocking updates. Two papers supply the event-driven design. asyTS in Parallelised Bayesian Optimisation via Thompson Sampling updates the posterior the moment any worker finishes. It outperformed its synchronous counterpart under random evaluation times. SPATS in Decentralized multi-agent active search for sparse signals has robots act on their latest local data without waiting for peers, giving roughly linear speedups.
Distinction from generative posterior sampling. Thompson Sampling via Fine-Tuning of LLMs samples candidates by fine-tuning a single model toward an external reward. We fine-tune nothing, and our "reward" comes from peer verification rather than an external oracle.
Delphi protocol elements. Anonymity and controlled statistical feedback are the defining features described in An international, Delphi consensus study to identify priorities for methodological research in behavioral trials in health research and in Conservative Management and Biological Treatment Strategies: Proceedings of the International Consensus Meeting on Cartilage Repair of the Ankle. Clinical Delphi studies apply item-level agreement statistics as retention thresholds, for example the content validity ratio in the VISION-Cog pilot. Our stopping rule borrows the logic of such thresholds but replaces raw agreement with posterior concentration.
A constraint on scope. CALMA shows that apparent agreement can hide divergent meanings. The protocol depends on clustering answers into hypotheses, so we restrict the main study to tasks where answer equivalence can be checked mechanically, such as mathematics and multiple choice. Open-ended deliberation of the kind discussed in AI Agents and the Future of Deliberation is out of scope until that clustering problem is addressed.
Research Program
We will build the coordinator and agent loop. An agent receives a problem, an assigned hypothesis, and an anonymized sample of prior rationales for and against that hypothesis. It returns a pass or fail verdict with a written justification. The agent may also propose a new hypothesis, which enters the belief model with a weak prior.
Heterogeneous latency will be simulated by drawing per-call delays from exponential and half-normal distributions, as in the asyTS experiments. We will also run a realistic setting that mixes fast and slow models.
Adversaries follow SAC's threat model: they broadcast false answers with confidence 1.0. We add a stronger variant that also lies on its verification verdicts.
We will analyze the adversarial bound formally. With per-agent cap $c$ and $F$ adversaries, the stopping rule cannot fire on a wrong hypothesis unless honest verdicts are themselves wrong at a rate we can state. We will measure that honest verdict error rate empirically.
Baselines
Naive single agent. A single agent answering once is the naive reference point.
Matched-budget majority vote. Independent sampling followed by a majority vote at the same total token budget is the "do nothing clever" panel. Panels must beat it to justify any coordination at all. We will run it with the same models and a temperature tuned on a held-out split, with the same budget given to every method.
Confidence-weighted averaging. CP-WBFT represents naive confidence-weighted consensus. SAC's authors report that its performance drops by as much as 24.8% on MATH500 Level 4 under a single adversary and often falls below 40% under attack.
SAC, the strongest existing method. SAC is the strongest competitor because it is the only protocol in the file designed for exactly this threat. Robust Multi-Agent LLMs under Byzantine Faults reports honest-majority accuracy up to 96.7% on MATH500 Level 4 and a Byzantine fault-tolerance improvement (BFTI) of up to 7.6%, using GPT-4o-mini and GPT-3.5-turbo. For Commonsense170k it gives no numerical breakdown; no reported result was found on Lacuna for that benchmark's accuracy. We will reimplement SAC on the same models, agent counts, communication graph, and $F$, and give it the same token budget by allowing extra rounds. Beating SAC means one of two outcomes. Either we get statistically higher accuracy under attack on the anchor-weakness subset (defined below) at no loss on the full set. Or we match its accuracy with significantly fewer tokens or less wall-clock time under heterogeneous latency.
Ablations of the key ingredients. Three ablations isolate what matters.
- Assignment rule. Uniform-random assignment tests whether sampling from the belief matters or whether any diversity suffices. Greedy assignment, which always verifies the current leader, is the deterministic UCB-style analogue.
- Asynchrony. A synchronous version of our protocol isolates the effect of asynchrony.
- Confidence exclusion. A version that feeds sender confidence back into the belief isolates the Delphi controlled-feedback rule.
Why not just…?
Why not just take a majority vote over independent samples? This is the most serious threat. On clean problems with no adversaries, it may well match any panel at equal compute. SAC's results show that aggregates blind to reasoning quality collapse under confident adversaries. However, that comparison was against confidence-weighted averaging, not plain voting. Plain voting has no confidence channel to exploit, so the clean and attacked settings must both be tested. If majority voting matches us under attack and in the anchor-weakness regime, our contribution reduces to latency savings, and we will say so.
Why not just run SAC for more rounds, or with a larger $F$? Larger $F$ discards more messages. That should worsen the anchor weakness, because wrong anchors throw away even more correct minority evidence. More rounds do not change what an anchor is willing to hear. Giving SAC a matched budget in our comparison tests this directly.
Why not just raise the temperature in ordinary multi-agent debate? Temperature produces diversity in wording, not targeted scrutiny of under-examined answers. The uniform-random assignment ablation is the fair version of this idea. If it matches Thompson assignment, the posterior adds nothing.
Why not just have everyone verify the current leading answer? Evidence from parallel bandits and concurrent reinforcement learning shows that deterministic argmax allocation duplicates effort. In our setting, a leader-only policy also never tests minority hypotheses, so a correct minority can never win. The greedy ablation measures this cost.
Proposed Experiments
We will use MATH500 Level 4 and Commonsense170k, following SAC. The models are GPT-4o-mini and GPT-3.5-turbo, plus an open Qwen model for reproducibility. We will vary panel size from 4 to 16, $F$ from 0 to 3, and the latency distribution.
The metrics are final accuracy, honest-majority accuracy, and BFTI as defined by SAC. We also report tokens and wall-clock time to stopping, and the rate of false stops.
Decisive check 1: the anchor-weakness subset. We will build a subset of problems where the honest agents' initial majority is wrong but at least one honest agent is right. SAC should fail systematically on these problems. Our protocol should recover the correct answer significantly more often, because the minority answer keeps receiving scrutiny in proportion to its posterior mass. If it does not, the core claim is false.
Decisive check 2: latency. Under heavy-tailed latency, the asynchronous version should reach the stopping threshold faster in wall-clock time than its synchronous twin, at equal accuracy. If the two finish at the same time, the asynchronous contribution is not supported.
Decisive check 3: verdict-lying adversaries. Against adversaries that also lie on their verification verdicts, accuracy should degrade gracefully as $F$ grows, staying within the bound implied by the influence cap $c$.
References
- Robust Multi-Agent LLMs under Byzantine Faults
- Parallel Contextual Bandits in Wireless Handover Optimization
- Society of Agents: Regret Bounds of Concurrent Thompson Sampling
- Parallelised Bayesian Optimisation via Thompson Sampling
- Decentralized multi-agent active search for sparse signals
- Thompson Sampling via Fine-Tuning of LLMs
- An international, Delphi consensus study to identify priorities for methodological research in behavioral trials in health research
- Conservative Management and Biological Treatment Strategies: Proceedings of the International Consensus Meeting on Cartilage Repair of the Ankle
- Feasibility, comprehensibility and acceptability of the VISION-Cog, a novel tool to assess cognitive impairment in visually impaired older adults: a cross-sectional pilot study in Singapore
- CALMA: Context‑Aligned Axes for Language Model Alignment
- AI Agents and the Future of Deliberation: Designing Human-AI Collaboration for Democratic Dialogue