What Happens at the Line: Boundary Rules as Hidden Decisions in Delphi Consensus

A Delphi study is a way of getting a panel of experts to agree without a meeting. Experts rate items anonymously, see a summary of the group's ratings, and rate again over several rounds. An item counts as "consensus" once agreement crosses a pre-set line, such as 67% in The Vascular Impairment of Cognition Classification Consensus Study. Every Delphi protocol also has to decide what to do with items that land at or just short of that line. It can drop them, carry them into the next round, or settle them by some fallback after the last round. Reports rarely treat this as a real choice. We think it is one, and that it matters more than its quiet treatment suggests. An item's fate in round one decides whether experts see it again and what group statistics they are shown. Those statistics in turn shape how they rate in round two. A boundary convention can therefore spread through the whole study and change items that were never near the line.

Computational social choice, the study of voting rules as algorithms, has already shown that tie-handling is not a footnote. In staged elections, switching between eliminating tied sub-election winners and promoting all of them changes how hard the election is to control (Anyone but him: The complexity of precluding an alternative). Swapping a random tied candidate for a randomly drawn ballot changes the complexity of manipulation for Borda and Bucklin (Ties Matter: Complexity of Manipulation when Tie-Breaking with a Random Vote). This proposal takes that insight from worst-case complexity into an empirical, outcome-level question about iterative consensus. We ask how much the final consensus set and its correctness depend on the boundary rule, and whether that dependence travels through the feedback loop. If it does, standard statistical checks on the final round cannot see it.

Core Claim

In threshold-based iterative consensus, the rule for items that fail to clear the line is an outcome-determining design parameter. Its effect is comparable in size to the choice between consensus and voting, and it works mainly through feedback cascades rather than through the boundary items themselves. Two consequences follow. First, fragility estimates computed by resampling final-round ratings will miss much of the real sensitivity, so a counterfactual "re-run under a different rule" measure is needed. Second, the coordinator who picks the rule has a form of agenda control that has not been measured.

There is a concrete hook in the multi-agent language-model literature. Voting or Consensus? Decision-Making in Multi-Agent Debate compares iterative consensus protocols with voting protocols for groups of Llama 3 agents. It reports 54.8% accuracy for voting versus 28.4% for consensus on MuSR, a multi-step reasoning benchmark built from murder-mystery puzzles. On MMLU, a broad multiple-choice knowledge test, consensus reaches 54.0% against 51.7% for voting. That paper treats the non-converged case as a loop or failure condition and does not study it. Our hypothesis is that a large share of the 26-point MuSR gap comes from what happens to questions stuck below the agreement threshold, not from consensus itself. If so, the reported split between consensus as a filter and voting as an explorer is partly an artifact of a boundary rule.

Research Program

The first component is a small formal model. A Delphi study is treated as a multi-stage elimination over items. Each round applies a threshold test with a strictness convention, meaning whether a score equal to the threshold passes. Items that fail are handled by a carryover rule, borrowed directly from the social-choice vocabulary. Under "ties-eliminate", failing items are dropped. Under "ties-promote", they return in the next round with feedback. Fallback rules decide items that are still unresolved after the last round. These are random candidate, random vote (one panelist's rating is drawn and decides, as in Ties Matter: Complexity of Manipulation when Tie-Breaking with a Random Vote), plurality, and default exclusion.

Two fragility quantities are defined on this model. Static fragility is the minimum number of individual rating changes in the observed final round that would flip the consensus set. It is computed exactly by integer programming on small instances. Dynamic fragility is the divergence between final consensus sets when the same panel and item pool are run from round one under different boundary rules. We also define coordinator reach. This is the number of items whose fate a coordinator could flip by choosing, after seeing round-one data, among rules that all look defensible on paper. It is a measurable analogue of the chair control formalized by Hemaspaandra et al.

We do not pursue worst-case hardness results. The random-vote study above found that in simulated elections the optimal manipulation under the two tie rules coincided about 98.7% of the time. That is a constraining result: it suggests worst-case hardness says little about practice, so our measures are empirical by design.

The second component is a set of counterfactual experiments. The same item pool and the same panel starting state are randomized across boundary-rule arms. For language-model panels this is cheap and exact: we fix seeds and initial independent drafts, then vary only the rule. For human panels it requires parallel matched panels.

Language-model panels carry a known bias. Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate reports that agents abandon correct answers at high rates in debate. Disagreement collapse in CommonsenseQA dropped from about 81% to 41% only once a designated judge was added. This bias motivates the human arm. Language-model panels are likely more conformist than human experts, so cascade sizes measured on them could be inflated. For that reason, every claim about magnitude is checked against human panels.

Proposed Experiments

The ground-truth testbed casts benchmark questions as Delphi items. On MMLU, GPQA, MuSR, StrategyQA and CommonsenseQA, a panel of agents rates candidate answers, and an answer is accepted once agreement crosses the threshold. Panels use Llama 3 8B and 70B, as in the Kaesberg et al. study, so that the comparison is direct. Thresholds of 2/3, 67%, 75% and 80% cover the 65–80% range common in Delphi practice. Each threshold is crossed with strict versus non-strict passing, the carryover rules, and the fallback rules. Because correct answers are known, we can measure accuracy as well as stability.

The applied testbed has two parts. The first re-analyzes published Delphi summaries for near-threshold mass. VICCCS, for example, reports 65% support for the SIVaD abbreviation against a 67% line. This part gives only static, summary-level estimates; raw round data would be requested from study authors and is not guaranteed. The second is a modest human Delphi with crowd panels. These panels rate factual and commonsense statements with known answers, and arms are randomized to ties-eliminate or ties-promote carryover.

The metrics are final accuracy, Jaccard divergence of consensus sets across rules, the disagreement collapse rate from Yao et al., coordinator reach, and the gap between dynamic and static fragility.

Baselines

The naive reference point is single-round independent majority voting with no feedback. Each agent drafts alone and the plurality answer wins. It shows whether iteration helps at all once boundary handling is controlled, and it is run with the same agents and models.

The strongest existing method is the pair of protocols in Voting or Consensus? Decision-Making in Multi-Agent Debate: iterative consensus at majority, supermajority and unanimity, and fixed-round voting. Both are run with All-Agents Drafting, which the authors report gives a 3.3% average accuracy boost. The reported figures are 54.0% consensus versus 51.7% voting on MMLU, and 54.8% voting versus 28.4% consensus on MuSR. For GPQA, StrategyQA and SQuAD 2.0, we found no isolated result on Lacuna. These protocols are the strongest competitor because they are the only published systematic comparison of threshold-based iteration against voting for agent panels.

We will rerun them ourselves rather than copy numbers. Model, agent count, maximum rounds and total token budget are matched across all arms, and prompts are tuned on a held-out split with the same budget for every arm. A boundary rule beats them if it exceeds the better of the two protocols on MuSR and stays within noise of consensus on MMLU at matched compute. That would show that one design captures both the "explorer" and "filter" advantages.

The statistical baseline for fragility is bootstrap resampling of final-round panelists, which gives confidence intervals on each item's agreement. Our method beats it if a meaningful fraction of items are stable under bootstrap yet flip across boundary rules in counterfactual re-runs.

The key ablation freezes feedback. Panels are shown no group statistics, or identical statistics, across arms. If rule effects shrink to roughly the boundary items alone, the cascade mechanism is confirmed. If they do not, the cascade claim is wrong.

Why not just…?

Why not just bootstrap the final-round ratings to get confidence intervals? This is the most serious alternative and is included as a baseline. Its limitation is structural. The final-round data were produced under one boundary rule, and resampling them cannot recover what panelists would have rated had different items returned with different feedback. The feedback-freeze ablation and the bootstrap-versus-counterfactual comparison settle the question. If bootstrap intervals already flag nearly every item that flips across rules, our dynamic measure adds little, and we would report that.

Why not just pre-register the threshold and rules? Pre-registration removes coordinator reach. It does not remove fragility, because a pre-registered ties-eliminate rule can still yield a less accurate or less stable consensus than ties-promote. Pre-registration also says nothing about which rule to register, and our accuracy results are meant to inform that choice.

Why not just use voting, given that it wins on reasoning tasks? This is a real threat. If ties-promote carryover with a plurality fallback performs no better than plain voting, then the consensus phase adds nothing beyond its filtering of easy items. The matched comparison against Kaesberg et al.'s voting protocol answers this directly. A null result would still be informative: it would locate the consensus/voting split entirely in boundary handling.

Decisive Checks

The proposal stands or falls on three outcomes. First, under matched compute, does changing only the fallback rule close most of the MuSR consensus–voting gap? Second, does dynamic fragility exceed bootstrap-detected fragility, with the excess vanishing when feedback is frozen? Third, do human crowd panels show a cascade in the same direction, even if smaller than in language-model panels? If the first two fail, boundary rules are a footnote after all. That would also be worth knowing for the many clinical Delphi studies that never report them.