When several language models deliberate over a question, they tend to fail in one of two opposite ways. Sometimes they converge on a confident answer that is wrong, or they confidently answer a question that has no answer. Other times they stall in permanent disagreement, or they retreat into empty "it depends" responses on questions that do have answers. Philosophers who write about epistemological double nihilism describe exactly this pair: rigid dogmatism on one side, and paralyzing doubt that treats every position as equally groundless on the other (Da Vinci versus Kafka, Bove on Vauvenargues). The Delphi method is the standard tool human institutions use when hard evidence is missing. Experts rate a question anonymously, see only the group's summary statistics and anonymized rationales, and re-rate over several rounds. It was designed to stop loud voices from dominating, but it still pushes every item toward a number.
This note proposes a Delphi panel for language models that can return one of three verdicts: "commit" to an answer, "no answer exists," or "genuinely contested, here are the positions." The central bet is that the panel should decide among these from how its disagreement changes across rounds, not from how much disagreement exists at the start. Under a rule that lets an agent change its rating only when it can point to a specific anonymous argument it had not already weighed, disagreement on a hard but answerable question should collapse because one side has a decisive reason. Disagreement on a controversial or ambiguous question should persist once the pool of new reasons runs dry. If this holds, the trajectory separates "hard" from "contested," which a single snapshot of disagreement cannot do.
Core Claim
Disagreement measured once among independent samples cannot tell a difficult answerable question from an irreducibly contested one, because both produce high spread. The claim is that disagreement measured over several anonymous, reason-gated rounds can tell them apart. Reason-gated means a rating change counts only if it cites an unseen rationale. The resulting panel should beat voting, debate, and judge-based systems on a joint objective. That objective penalizes dogmatism, meaning committing on unanswerable or contested items or reaching wrong consensus. It equally penalizes paralysis, meaning abstaining or declaring contest on answerable items.
The philosophical literature plays a motivating role only. It names the two failure modes and argues for an "active" skepticism that keeps inquiring rather than suspending judgment (Comment peut-on être sceptique?). Educational work shows that unchecked relativism erodes the willingness to take evidence-based stances (English majors' critical thinking). None of this work offers a procedure. What we take from it is the evaluation principle: both failures count, and a system that avoids one by falling into the other has not succeeded.
Relation to Prior Work
The closest work is An LLM-based Delphi Study to Predict GenAI Evolution. It runs a five-round automated Delphi with an organizing agent and persona-assigned responders, all built on GPT-4o-mini, and reports thematic consistency across 12 configurations. It has no ground truth and no abstention or contest outcome, and it reports an uncontrolled positivity bias. We keep its facilitator design but evaluate on labeled answerable, unanswerable, and contested items.
Three papers constrain the design by showing that iteration itself is dangerous.
-
Voting or Consensus? finds that consensus beats voting on knowledge tasks (54.0% vs 51.7% on MMLU), loses badly on reasoning tasks (28.4% vs 54.8% on MuSR), and drifts with more rounds. It reports, without a number, that consensus better identifies unanswerable SQuAD 2.0 questions. Its All-Agents Drafting (independent first answers before seeing peers) is our round 1. It selects a protocol statically, whereas we let round dynamics choose the verdict. - Relational Priors as Convergence Pressure shows that social framing raises wrong consensus on MMLU-Pro from 11.2% to 32.0%. It also shows that even an explicit "be neutral" instruction hurts relative to no prompt. This is why our protocol strips identities and social language entirely instead of prompting for virtue.
-
Peacemaker or Troublemaker ties sycophancy to abandoning correct answers (r = 0.902) and shows that fixed independent personas deadlock while fixed agreeable ones collapse. That is double nihilism in miniature, produced by static personas. Our reason-gating is meant to replace static stance with stance that depends on arguments.
Two papers supply partial mechanisms we do not adopt.
- CONSENSAGENT detects stalled or sycophantic debates and rewrites the prompt. It treats stalls as failures to repair, whereas we treat some stalls as the correct "contested" verdict.
- MADISSE forces opposing initial stances on summary faithfulness and catalogues 16 ambiguity types. Filtering ambiguous cases raises human–model agreement from α = 0.53 to 0.71, which is direct evidence that ambiguous items exist and should be routed differently. It still forces binary stances, while we let the panel output ambiguity.
DELPHI supplies human-labeled controversial questions and two metrics: controversy acknowledgement and comprehensiveness. It has only ever been evaluated single-turn. Finally, AI Agents and the Future of Deliberation and CALMA warn that synthetic deliberation can impose one model's values on plural questions. That warning is a reason the contested verdict must report positions rather than adjudicate them.
Research Program
Round 1. A panel of N agents (default N = 7) each answers independently. Every agent gives three things: a categorical answer that may include "no answer exists," a 1–9 confidence rating, and a short rationale.
Feedback. A facilitator returns the distribution of answers to each agent, as counts, median confidence, and interquartile range. It also returns the rationales, deduplicated by embedding similarity, labeled with arbitrary IDs, and stripped of any identity, persona, or count of supporters. This keeps Delphi's anonymity and controlled feedback while removing the headcount-per-argument signal that drives conformity.
Later rounds. Each agent may keep its answer, or change it by naming the rationale ID that moved it and stating what that rationale shows that its own did not. A separate checker model verifies three conditions: the cited ID exists, it was not authored by the same agent, and the stated content actually appears in it. Changes that fail the check are reverted. This reason-gate is the operational form of "active skepticism": doubt is allowed to move a position only through an argument, never through the tally.
Stopping and verdict. The panel stops when a round produces no new verified changes and no new rationale that survives deduplication, or after a cap of five rounds. The verdict is then assigned as follows:
- Commit: a single answer holds a supermajority, and the shift toward it was reason-backed.
- No answer exists: that option holds the supermajority.
- Contested: dispersion remains above threshold after the reason pool is exhausted. The panel outputs the surviving positions with their strongest rationales.
The supermajority and dispersion thresholds are tuned once on a held-out development split and then frozen.
Models and analysis. We will run panels drawn from a single open-weight model family (Llama 3.3-70B, Qwen3-32B). We will also run mixed-architecture panels, to test the homophily concern left open by the GenAI Delphi study. The main analysis is descriptive before it is comparative. We will plot, per item type, dispersion across rounds and the fraction of changes that were reason-verified. The hypothesis predicts falling dispersion with high verified-change rates on answerable items, and flat dispersion with few verified changes on contested items.
Baselines
Naive reference. A single model with chain-of-thought prompting is instructed to answer, say "no answer," or say "contested." This shows whether a panel is needed at all.
Independent-dispersion self-consistency. This is the strongest cheap competitor. We draw N×R independent samples, matching the panel's total generation count, and read the verdict from the round-1 answer distribution alone, using the same frozen-threshold tuning budget. It is the cleanest test of whether the rounds add anything.
Voting and consensus protocols. These come from Voting or Consensus?, run with their All-Agents Drafting. We run its best reasoning protocol (voting) and its best knowledge protocol (consensus), each given the same abstain option. Its reported numbers are 54.0% (consensus, MMLU) and 54.8% (voting, MuSR) accuracy. Our three-way setting has no reported comparator, so all of these baselines will be rerun under identical models and budgets.
Unblinded debate. This is debate with no relational prompt (the NO-PRIOR condition from Relational Priors), plus a centralized-judge variant following Peacemaker or Troublemaker. That paper reports the judge cutting Disagreement Collapse Rate from about 81% to 41% on CommonsenseQA.
Repair-based debate. This is CONSENSAGENT, which reports 0.62 vs 0.43 accuracy over plain debate on CLUTRR. It is run where its fine-tuned optimizer can be reproduced; otherwise we use a prompted stand-in, clearly flagged as such.
Ablations. Two ablations isolate the key ingredients:
- Remove the reason-gate while keeping anonymity, to isolate gating.
- Restore identities and transcripts while keeping the gate, to isolate anonymity.
Fairness and win condition. Every system gets the same models, temperature, item splits, total-token budget, and threshold-tuning budget. Beating a baseline means higher three-way macro-F1 with no increase in either the dogmatism rate or the paralysis rate, with bootstrap confidence intervals over items.
Why not just…?
Why not just abstain when independent samples disagree? This is the real threat. If round-1 dispersion already separates contested from answerable items as well as the full trajectory does, the proposal collapses to self-consistency. MuSR gives a reason to doubt it. Hard answerable items there produce wide disagreement, which is exactly why consensus scores only 28.4% where voting reaches 54.8% in Voting or Consensus?. Snapshot dispersion should therefore mislabel many of them as contested. The compute-matched comparison above settles this question directly, and we commit to reporting a null result if the trajectory adds less than the confidence interval.
Why not just use a strong judge? A judge does reduce collapse (Peacemaker or Troublemaker). But the same paper documents judges rewarding confident style. On DELPHI-type questions a judge must also pick a side, or else hedge generically. GPT-3.5's 84.3% acknowledgement rate came largely from boilerplate disclaimers (DELPHI). The comparison on contested items, scored for acknowledgement and comprehensiveness together, will show whether a judge can map positions as well as a panel.
Why not just run plain debate without any special prompting? NO-PRIOR is indeed the best of the social-prompt conditions in Relational Priors. However, plain debate still shows agents abandoning correct answers after rounds 2–3, and it drifts with more rounds. Our anonymity ablation keeps everything else fixed and tests whether removing peer identity and transcripts lowers wrong-consensus rate relative to NO-PRIOR debate.
Proposed Experiments
Evaluation set. We assemble a three-way labeled set:
- Answerable: MMLU-Pro, MuSR, and answerable SQuAD 2.0 items.
- Unanswerable: SQuAD 2.0 unanswerable items (Rajpurkar et al.). The roughly 7% annotation noise reported there among unanswerables is handled by adjudicating a subsample.
- Contested: DELPHI's 2,281 highly controversial questions, plus AggreFact and TofuEval summaries that MADISSE's taxonomy marks ambiguous. Low-controversy DELPHI questions serve as answerable-but-opinion-adjacent controls.
Metrics.
- Three-way macro-F1.
- Dogmatism rate: commits on unanswerable or contested items, plus Wrong Consensus Rate.
- Paralysis rate: abstain or contested verdicts on answerable items.
- Accuracy on committed items, together with coverage.
- Disagreement Collapse Rate.
- On contested items, DELPHI acknowledgement and comprehensiveness, plus a human rating of whether the reported positions span the views annotators identified.
For calibration of expectations, there is no reported result on Lacuna for any multi-agent system on DELPHI, nor a reported number for consensus abstention on SQuAD 2.0.
Decisive checks. Four results would decide the question:
- Trajectory beats snapshot. The full trajectory must beat compute-matched round-1 dispersion on separating MuSR-hard items from DELPHI-contested items, measured by AUROC of the contested-vs-answerable decision. If it does not, the core claim fails.
- Gating avoids both failures. Removing the reason-gate should raise wrong consensus without lowering paralysis. If instead gating only buys lower dogmatism at the cost of more paralysis, we have traded one nihilism for the other.
- No drift. Accuracy on committed answerable items should not decline across rounds, unlike the drift reported for plain consensus.
- Plural positions survive. Mixed-architecture panels should report contested verdicts on DELPHI more often than single-family panels without paying for it in paralysis on low-controversy controls. That is evidence the contested verdict reflects real plurality rather than one model's hedging.
References
- Da Vinci versus Kafka: Looking for Answers
- Laurent Bove, Vauvenargues ou le séditieux. Entre Pascal et Spinoza. Une philosophie pour la seconde nature
- Comment peut-on être sceptique? Hommage à Didier Deleule. Études réunies par Michèle Cohen-Halimi et Hélène L’Heuillet
- A Study on the Current Situation of English Majors’ Critical Thinking Skills in the Context of New Liberal Arts Construction
- An LLM-based Delphi Study to Predict GenAI Evolution
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- Relational Priors as Convergence Pressure in LLM-Based Multi-Agent Systems
- Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate
- CONSENSAGENT: Towards Efficient and Effective Consensus in Multi-Agent LLM Interactions Through Sycophancy Mitigation
- Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation
- DELPHI: Data for Evaluating LLMs' Performance in Handling Controversial Issues
- AI Agents and the Future of Deliberation: Designing Human-AI Collaboration for Democratic Dialogue
- CALMA: Context‑Aligned Axes for Language Model Alignment
- Know What You Don't Know: Unanswerable Questions for SQuAD