The Delphi method gathers expert judgment anonymously over several rounds: each expert rates a set of statements, sees a summary of the group's answers, and revises. The method mostly feeds back numbers, such as a median and a spread. That throws away the "it depends" structure of expert reasoning. Two clinicians who rate "start drug X" as 3 and 8 may not disagree at all. One may be picturing frail elderly patients and the other young adults. A median turns this difference in scope into apparent disagreement. It can also hide real disagreement inside an apparent average.
We propose a different feedback object. We convert each expert's written rationale into a small conditional tree (if the patient is frail, then low; otherwise, if the dose is adjustable, then high). We then build one shared diagram for the whole panel by merging every subtree that is identical across experts. In the merged diagram, shared reasoning appears once and is labelled with how many experts hold it. Divergence sits exactly at the branch where experts part ways. Consensus can then be scored per context rather than per statement. Panelists see where they truly differ, as opposed to cases where they simply covered different situations.
A second ingredient is a check borrowed from Ramsey theory, the branch of combinatorics showing that large structures always contain some orderly substructure. It keeps the merged diagram from reporting agreement that would appear in any panel of that size whatever its members believed.
Core Claim
Exact merging of identical conditional subtrees, after expert language has been normalized to a shared vocabulary of conditions, yields a deterministic and auditable consensus diagram. Feeding this diagram back between Delphi rounds should do three things. It should separate genuine divergence from mismatches in which experts covered different situations. It should reduce false convergence in AI-simulated panels. It should automate the step that currently needs a skilled human facilitator.
The claim is testable against two reference points. The first is median feedback, the classical Delphi summary. The second is a large language model (LLM) writing a thematic summary of rationales, which is the strongest cheap competitor. If the LLM summary does as well on every outcome, the contribution shrinks to auditability alone, and we will report that.
What Prior Work Does and Leaves Open
The motivation comes from two Delphi-side papers. The Human-AI Hybrid Delphi Model diagnoses "interpretive flattening", the loss of context when answers are reduced to a scalar. It responds with a four-tier classification (Strong, Conditional, Operational, No Consensus) and a seven-category coding scheme for rationales. Both depend on a skilled human facilitator, and the authors identify that facilitator as the bottleneck to scaling. Our merged diagram is an algorithmic version of their "Conditional Consensus" tier: agreement holds within each context while differing across contexts.
DEBATE shows that zero-shot LLM panels collapse toward premature agreement. Human groups show almost no change in opinion spread (ΔSD ≈ 0), while LLM panels shrink sharply (ΔSD ≪ 0). Should we be going MAD? finds that multi-agent debate is fragile with respect to round count and agent count, costs five to fifteen times more than a single agent, and is best tamed by prompting agents to agree a fixed share of the time. Both papers motivate feedback that is structural rather than conversational.
The merging method comes from node-merging work. On the Two Sides of Redundancy in Graph Neural Networks canonizes neighborhood trees and collapses isomorphic subtrees into a single directed acyclic graph (DAG) in polynomial time. We reuse that canonize-then-hash idea on rationale trees rather than on graph neighborhoods. Oblivious Decision Trees, Graphs, and Top-Down Pruning shows that merging compatible tree nodes both compresses models (158.4 to 17.2 nodes on German credit) and pools sparse data. That second effect matters here, because a single expert's branch is a sample of one. Compact Belief State Representation for Task Planning shows that And-Or DAGs separating independent sub-conditions beat Binary Decision Diagrams, which depend on choosing a good variable order. We use its AND/OR node types so that experts who list the same conditions in different orders still merge.
The closest formal precedent is Merging Argumentation Systems. It merges agents' attack graphs over atomic arguments by minimizing edit distance, and it shows that merging structure beats voting on outcomes. It treats arguments as given atoms with no internal conditions. It also cannot express that two experts covered different situations rather than conflicting. Its majority operator can create cycles. We inherit its lesson that structure should be merged rather than outcomes voted on. We differ by merging conditional content, which makes scope visible, and by handling cycles with the unavoidability check below instead of ignoring them.
Building the Merged Diagram
In each round, every panelist gives a rating and a free-text rationale per item. An LLM extractor maps each rationale onto condition literals, such as "patient frail" or "dose adjustable", drawn from a shared vocabulary that grows each round. New literals are clustered against existing ones. A facilitator can veto a cluster in seconds, which is far less work than coding themes. The extractor then outputs an AND/OR tree whose leaves carry the panelist's rating for that context.
Trees are canonized bottom-up, with children sorted under AND and OR. Identical canonical subtrees are then merged into one node that records which panelists support it. Merging is exact. The only approximate step is normalizing literals, so every merge can be traced to a vocabulary decision that a person can inspect.
For each root-to-leaf context, we compare ratings among the panelists whose trees cover that context. Each item then falls into one of four classes:
- Unconditional agreement: panelists agree everywhere.
- Conditional agreement: panelists agree within each context, but ratings differ between contexts.
- Scope mismatch: panelists rated disjoint contexts. This is new information that the next round's questionnaire asks the missing panelists to fill in.
- Genuine divergence: panelists covering the same context gave conflicting ratings.
The Unavoidable-Agreement Check
Ramsey theory says that sufficiently large structures cannot avoid orderly pieces. Unavoidable structures in infinite tournaments characterizes which acyclic patterns embed in every tournament, a tournament being a complete set of pairwise "A beats B" verdicts. No Cliques Allowed uses Ramsey's theorem to show that large tournaments force a cycle in bounded-depth rule systems. We use the finite, elementary consequence: every tournament on $m$ options contains a consistent sub-ranking of at least $\lfloor \log_2 m \rfloor + 1$ options. This is the classical Ramsey-type bound for tournaments.
This matters for Delphi prioritization rounds. Within each merged context, the majority's pairwise preferences form a tournament, which may contain cycles, as the argumentation-merging work warns. We report the longest consistent chain. We count it as consensus only if it exceeds both the worst-case floor and an empirical null in which panelists' rationales are shuffled across items.
The same shuffled null bounds the merge rate itself. Any merging that also occurs on shuffled rationales reflects vocabulary coarseness rather than shared reasoning. This is a guard rail, not a theorem about deliberation. Its job is to stop the diagram from manufacturing the over-convergence it is meant to fix.
Research Program
The work has three parts.
Merge fidelity. We build and validate the extractor and merger. Human annotators judge whether merged nodes truly express the same reasoning. We measure precision of merges, compression ratio, and how stable the vocabulary stays across rounds.
Simulated panels. We test whether diagram feedback changes the dynamics of LLM-simulated panels relative to the alternatives. The main question is whether it reduces over-convergence without hurting accuracy.
Prospective human Delphi. We run a small two-arm study with real experts, where the facilitator's time and the panel's judgment of the feedback can be measured directly.
Baselines
Median-and-spread feedback. The naive reference point is classical Delphi feedback: median, interquartile range, and the standard 70%-agreement consensus rule, with no rationales shown. It runs on identical items, panels and round counts, and every other condition must beat it on the outcomes below.
LLM thematic summary. This is the strongest competitor and the one a skeptic would pick: an HAH-Delphi-style facilitator, following The Human-AI Hybrid Delphi Model. The same LLM that runs our extractor instead writes an anonymized thematic summary of the rationales, prompted with the seven-category scheme and the four-tier labels. Using the same model, the same rationales and a matched token budget isolates the representation as the only difference. On historical consensus studies, that paper reports 95% replication of consensus items and 95% directional agreement with six senior sleep experts on a 20-item insomnia questionnaire. Our method beats this baseline only if it matches those agreement levels and also does at least one of two things. It must either recover significantly more conditional-consensus items that the human panel later endorses, or keep simulated-panel spread closer to the human value.
Agreement-modulated debate. We include Should we be going MAD?'s 90% agreement-modulated Multi-Persona debate. It is given the same agents and rounds, plus a tuning grid over the agreement level equal in size to our grid over vocabulary thresholds. That paper reports a best-in-class GPT-3.5 MedQA result but no accuracy figure appears in the file, so: no reported result found on Lacuna for MedQA accuracy of agreement-modulated debate.
Sum-operator argumentation merge. We include the Sum-operator merge from Merging Argumentation Systems, applied to arguments extracted by the same LLM. This tests whether conditional content matters beyond attack structure.
Ablations. Two ablations isolate the key ingredient. The first shows every panelist's tree with no merging, testing whether merging itself, rather than structured display, does the work. The second drops the unavoidable-agreement check, testing whether the guard rail prevents spurious consensus.
Why not just…?
Why not just let an LLM summarize the rationales? This is the real threat. LLM summaries may already convey the "it depends" structure well enough. Prior evidence cuts both ways. DEBATE shows that free-text LLM interaction drifts toward agreement, which suggests a summary may smooth over divergence. HAH-Delphi, however, shows that LLM output tracks expert consensus closely. The matched-budget comparison settles the question. If summaries tie on the spread of opinion and on recovering conditional consensus, we will report that the diagram's value is auditability and the scope-mismatch signal, not better outcomes.
Why not just prompt agents to agree less or more? Agreement modulation tunes a single global level and is reported to fluctuate with round count and agent count. It cannot tell a panel where the disagreement is. Our simulated-panel experiment sweeps rounds and agent counts for both methods. The diagram should show flatter sensitivity. If it does not, this objection stands.
Why not just merge argument graphs as in the argumentation literature? Atomic arguments cannot encode that two experts addressed different patients. The Sum merge would then count a scope mismatch as an attack. The comparison counts how many items each method labels divergent that the human panel later resolves as scope differences.
Why not just average more panelists? Adding panelists shrinks the variance of the median. It does not recover conditions that the median discarded. More panelists also inflate the agreement that the Ramsey floor and shuffled null show to be unavoidable.
Proposed Experiments and Decisive Checks
Simulated panels on DEBATE. We use DEBATE's 107 topics. LLM panels are seeded with the human participants' initial stances and rationales, then run four Delphi rounds under each feedback condition. We use Llama-3.1-8B-Instruct, the model whose figures DEBATE reports, without fine-tuning. Metrics are DEBATE's change in opinion spread (ΔSD), with humans at ≈0, and DEBATE's stance error.
DEBATE reports that supervised fine-tuning lowered Llama-3.1-8B-Instruct stance error from 1.54 to 1.29. The decisive check is whether diagram feedback alone, with no training, gives stance error below 1.29 and ΔSD significantly closer to zero than both median feedback and LLM summaries.
Accuracy on MedQA. We run the same panels on MedQA. Final answer accuracy and API cost are measured against agreement-modulated debate. The goal is to confirm that preserving dissent does not cost accuracy and that cost stays below the five-to-fifteen-fold overhead of debate.
Prospective human Delphi. Two arms of six to twelve clinicians, matching HAH-Delphi's saturation size, rate one clinical questionnaire. One arm receives LLM-summary feedback and the other receives diagram feedback. Outcomes are:
- how many items end as conditional consensus;
- how many disagreements panelists later label as mismatches in which they had considered different situations;
- facilitator minutes per round;
- panelists' blinded ratings of how faithfully the feedback represents them.
Merge fidelity. Annotators judge whether merged nodes truly express the same reasoning. We require merge precision above 0.9, and a merge rate on shuffled rationales well below the real rate. Without both, the diagram cannot be trusted as evidence of shared reasoning.
References
- The Human-AI Hybrid Delphi Model: A Structured Framework for Context-Rich, Expert Consensus in Complex Domains
- DEBATE: A Large-Scale Benchmark for Role-Playing LLM Agents in Multi-Agent, Long-Form Debates
- Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
- On the Two Sides of Redundancy in Graph Neural Networks
- Oblivious Decision Trees, Graphs, and Top-Down Pruning
- Compact Belief State Representation for Task Planning
- Merging Argumentation Systems
- Unavoidable structures in infinite tournaments
- No Cliques Allowed: The Next Step Towards BDD/FC Conjecture (Extended Abstract)