Quality-Precedent Feedback for Delphi Loops

The Delphi method gathers expert judgment in rounds. Experts answer independently, a coordinator summarizes the panel's answers, and everyone revises after seeing the summary. The same loop now runs inside multi-agent debate between language models, and it fails there in a familiar way: after two or three rounds the agents cave to whatever the majority says, including when the majority is wrong. This note argues that the failure comes mostly from the summary that is fed back between rounds, not from the rule that makes the final decision. Every standard summary counts heads. Medians, agreement thresholds and "show the most common rationales" all do this, and argumentation theory has proven that counting heads cannot be reconciled with letting the single strongest argument govern. We propose to choose what gets fed back using a ranking with an axiomatic guarantee that it favors quality over volume and that still behaves well on the cyclic attack-and-support graphs that real deliberation produces.

In two sentences: treat each Delphi round as a graph in which panelists' rationales support or attack candidate answers and each other, and feed back the rationales that a "quality-first" graph ranking scores highest instead of the popular ones. We then prove which ranking rules can do this on cyclic graphs and test whether the choice of feedback rule, holding the panel and the final vote fixed, predicts whether a correct minority survives or collapses.

Core Claim

Two lines of axiomatic work meet here without having been connected. Argumentation theory gives the obstruction. Evaluation of Arguments from Support Relations: Axioms and Semantics proves that on support graphs no strength function satisfies both Cardinality Precedence and Quality Precedence. Cardinality Precedence says more supporters means stronger. Quality Precedence says the best supporter governs. A Comparative Study of Ranking-Based Semantics for Abstract Argumentation proves the same conflict for attackers. Network-centrality axiomatics supplies the tool for cycles. An Axiomatization of the Eigenvector and Katz Centralities shows that Katz centrality's constant baseline score is exactly what buys component-level locality, and Random Walk Decay Centrality uses locality-style axioms to keep rankings stable on graphs where PageRank misbehaves. The best-behaved bipolar argument semantics, the Euler-based semantics of Weighted Bipolar Argumentation Graphs: Axioms and Semantics, is proven sound only on acyclic graphs. Deliberation graphs are cyclic by nature.

The claim has two parts. The theoretical part is that on cyclic signed graphs, a baseline-damped, top-based feedback ranking satisfies Quality Precedence, a signed analogue of Set Locality, and convergence, while every linear signed Katz-style ranking is forced into compensation and therefore violates Quality Precedence. The empirical part is that within a fixed multi-agent Delphi protocol, the feedback rule's position on this trade-off predicts minority survival and disagreement collapse, and the final decision rule matters much less.

Research Program

Delphi round graph. A Delphi round produces a set of candidate answers and a set of short rationales, one or more per panelist. A separate extraction model labels each ordered pair of rationales, and each rationale-answer pair, as support, attack or unrelated. This yields a weighted bipolar graph. Every node carries an intrinsic weight $w(a)\in[0,1]$, which we set uniformly as the default, as panelist-stated confidence in a variant, or as an evidence-grounding score in a variant. Self-reinforcing clusters and mutual rebuttals make the graph cyclic.

Ranking rules compared. We compare four semantics on the same graph:

  1. A cardinality rule in the spirit of the reward-based semantics, with diminishing returns per extra supporter.
  2. A linear compensating rule, which is signed Katz.
  3. The Euler-based semantics, with cycles broken arbitrarily.
  4. The proposed damped top-based rule.

The proposed rule's strength $s(a)$ for each rationale or answer $a$ is the fixed point of

$$s(a) = (1-\delta)\,w(a) + \delta\Big[\max_{b \in S(a)} s(b) - \max_{c \in A(a)} s(c)\Big]_{[-w(a),\,1-w(a)]},$$

where $S(a)$ and $A(a)$ are $a$'s supporters and attackers, $\delta\in(0,1)$ is a damping factor, and the bracket clips the update so that $s(a)$ stays in $[0,1]$. The $(1-\delta)w(a)$ term plays the role of Katz's baseline. Because max, min and clipping are 1-Lipschitz in the sup norm, the update is a $\delta$-contraction. That gives a unique fixed point on arbitrary cycles, which is the property the Euler-based semantics lacks.

Theorems. We aim for three results, from easiest to hardest:

  1. Feedback-budget impossibility. Suppose only $k$ rationales can be shown per round. Any anonymous feedback rule satisfying Cardinality Precedence can be made to exclude every single-supporter rationale by adding enough generic supporters to $k$ rivals. No quality-precedent rule can be made to do this. This is close to a corollary of the Amgoud–Ben-Naim result, but stated for the operator that Delphi actually iterates.
  2. Characterization. On cyclic signed graphs, the rules satisfying signed Endpoint Removal, signed Set Locality, Quality Precedence and contraction-continuity are exactly the damped top-based family, up to monotone reparametrization. The proof would adapt the "unity graph" induction of Was and Skibski, since graphs in which every node has at most one incoming edge make max-aggregation and sum-aggregation coincide.
  3. Linear compensation. Every linear signed feedback centrality violates Quality Precedence.

Theorem 2 is the risky one. If uniqueness fails, we will report which axiom must be added or weakened.

Iterated protocol. Each round, every agent sees the top-$k$ rationales under the chosen rule, with no vote counts attached. Rounds stop at three, following the drift findings below. A fixed final rule then picks the answer, so that the feedback rule is the only thing that varies. Following Germane Conflicts: Desirable Properties for Localising Inconsistency, we deliberately do not try to localize the background premises behind disagreements, because doing that check exactly is $\Sigma^p_2$-hard. We stay at the level of rationale graphs.

Baselines

The naive reference point is single-round independent answering followed by majority vote, with no feedback at all. It measures what deliberation adds.

The strongest existing method on our main benchmark is the voting protocol of Voting or Consensus? Decision-Making in Multi-Agent Debate. That paper compares consensus-until-agreement against fixed-round debate followed by voting with Llama 3, and reports 54.8% accuracy for voting on MuSR against 28.4% for consensus. We rerun their voting protocol, together with their All-Agents Drafting variant, which they report adds 3.3% on average. We use the same Llama 3 70B agents, the same number of agents, the same three rounds and the same token budget per round. That way the only difference from our method is which rationales each agent sees between rounds. Beating it means a statistically significant gain on MuSR at matched compute, and in particular on instances where a correct answer starts out as a minority.

For collapse, the strongest reference is the central-judge architecture of Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate, which reports cutting the Disagreement Collapse Rate on CommonsenseQA from roughly 81% to 41%. We run it with the same backbone and measure collapse with their metrics. Beating it means lower collapse without a lower correct-answer rate.

A Delphi-faithful baseline shows the panel's median and a random sample of anonymous rationales, which is classical controlled feedback. Three ablations isolate our key ingredient while holding the graph and extraction fixed:

  1. Swap the top-based rule for the cardinality rule.
  2. Swap it for signed Katz.
  3. Swap it for the Euler-based semantics with cycles broken.

If these ablations perform equally well, the quality-precedence axiom is not what helps.

Why not just…?

Why not just take a majority or Borda vote after the last round? This changes only the final decision rule. Our hypothesis is that the damage happens earlier, when popular rationales crowd out correct minority ones in the feedback. On the Aggregation of Argumentation Frameworks also shows that majority-based aggregation of debate structure fails most rationality tests. The experiment settles the question directly: we hold the final vote fixed and vary only the feedback rule. If minority survival does not change, the proposal is wrong.

Why not just weight votes or rationales by stated confidence? This is the most serious threat. Confidence weighting is a cheap approximation of quality precedence, and it needs no graph extraction. We include it twice: as a feedback-selection baseline, and as the intrinsic weight inside our own rule. If confidence-ranked feedback matches the top-based rule, then the graph structure adds nothing and we will say so. Our expectation is that sycophantic agents report inflated confidence, so confidence tracks agreeableness, given the 0.902 sycophancy-error correlation in the Peacemaker paper. But this is an empirical bet, not a known result.

Why not just run PageRank or signed Katz on the endorsement graph? Unsigned centralities treat a rebuttal as an endorsement, and the Was–Skibski analysis covers only unipolar graphs. Signed Katz fixes the sign problem but is linear and therefore compensatory, which is exactly the Cardinality-side failure we aim to avoid. It is one of our ablations, so its performance is measured rather than assumed.

Why not just have an LLM facilitator pick the most important rationales? The Human-AI Hybrid Delphi Model shows that Gemini 2.5 Pro reaches 95% directional agreement with senior experts. But its authors still rely on a human facilitator for synthesis, and judges can themselves be sycophantic. We run an LLM-selector baseline with the same backbone and budget. It offers no guarantee about volume-driven selection, and we test whether it behaves like a cardinality rule on instances where the planted minority is correct.

Proposed Experiments

The main benchmark is MuSR (MuSR): 756 multistep narrative reasoning instances, where voting beats consensus by a wide margin. We report accuracy per domain and minority-recovery rate. Minority recovery is the fraction of instances where the correct answer held by at most two of five agents in round one is the final answer.

The decisive controlled test plants minorities. We seed one agent with a correct, well-grounded rationale and four agents with a shared incorrect answer and generic rationales. We then vary the number of generic supporters from 1 to 8. The theory predicts that cardinality and signed-Katz feedback lose the minority once the supporter count crosses a threshold. It predicts that top-based feedback does not, unless a rationale directly attacks the minority one with higher intrinsic weight.

For collapse, we use MMLU-Pro and CommonsenseQA and report Disagreement Collapse Rate and Negative Agreement Rate. We also run MMLU, where consensus beat voting (54.0% vs. 51.7% in Kaesberg et al.), to check that quality-first feedback does not lose the error-filtering benefit on knowledge tasks.

Graph extraction is a known weak point. We report extraction agreement against human labels on 200 sampled rationale pairs, and we rerun the main comparison with 10–30% of edge labels randomly flipped.

The proposal fails if any of three things happens:

  1. The top-based rule does not beat the cardinality ablation on planted minorities.
  2. Confidence-ranked feedback matches it.
  3. Its advantage disappears under realistic extraction noise.