The Delphi method is how medicine, public health and standards bodies turn a panel of experts into a recommendation. Experts answer anonymously. A coordinator shows everyone a summary of the panel's answers, and the experts revise over several rounds. That summary is almost always a median, an interquartile range and a few hand-picked written justifications, so the feedback pulls every round toward the middle. When a panel "reaches consensus," nobody can tell whether the minority was persuaded or simply stopped seeing its own view reflected back. This note proposes treating the feedback step itself as a committee-selection problem. Out of hundreds of anonymous justifications, a small set is chosen to show the panel, and it is chosen with a greedy proportional voting rule from social choice. Such rules guarantee that any sufficiently large, like-minded subgroup gets at least one statement it endorses.
The second half of the proposal is a measurement design. It asks whether convergence under proportional feedback is more genuine than convergence under median feedback, using a no-feedback control arm and delayed private re-elicitation. These tests separate real persuasion from what the interface literature calls deference: accepting what is shown because contesting it is costly. In two sentences: Delphi feedback is a slate of statements, and it should be chosen so that every sizable faction sees itself in it. We then test whether this keeps legitimate dissent alive without blocking real agreement, using human-centric outcome measures rather than consensus rates.
Core Claim
The claim has three parts. First, a word-budgeted greedy proportional rule (sequential Phragmén or the Method of Equal Shares) applied to panelists' own anonymized rationales keeps cohesive minorities represented in each round's feedback more reliably than median-plus-rationales, top-approval or LLM-summary feedback. Second, it does so without reducing the convergence that panelists still endorse when asked privately afterwards. Third, the gap between "displayed convergence" and "endorsed convergence" is a measurable quantity that standard Delphi reporting hides.
The new element is not proportional slate selection, which exists. It is putting that selection inside a multi-round loop where the slate changes the preferences it is supposed to represent, and evaluating that loop by its effect on people.
Relation to Prior Work
The method is supplied by generative social choice. Generative Social Choice defines Balanced Justified Representation and an LLM oracle that predicts how a participant would rate a statement. It validated a one-shot slate on US abortion opinions, where 84 of 100 participants in a second cohort felt "excellently" or "exceptionally" represented. Generative Social Choice: The Next Generation adds a word-count budget and noisy-oracle guarantees in PROSE. That system's greedy DemocraticProcess beats clustering and zero-shot baselines by 10–40% in mean satisfaction, by 40% in bottom-quartile utility, and has 7–49 percentage points fewer BJR violations on Bowling Green and UCI Drug Review data. Both are one-shot batch pipelines. Participants write once and receive a final slate, and nobody measures how a slate shown mid-deliberation shifts the next round. We reuse their oracle and budget model but change the object of study: the slate is an intervention, not an output.
The greedy rules themselves come from multiwinner voting. In these rules each selected statement "spends" the voting weight of the voters who approve it, so a majority cannot take every seat. Phragmén's Voting Methods and Justified Representation shows sequential Phragmén guarantees Proportional Justified Representation in polynomial time. Proportional Justified Representation shows that reweighted approval voting loses even basic Justified Representation once the committee size $k \ge 6$. We therefore do not use reweighted approval as our main rule. An Experimental Comparison of Multiwinner Voting Rules on Approval Elections finds that greedy proportional rules behave like their exact counterparts in practice, which justifies avoiding exact solvers.
Two lines of work motivate the concern. Delphi practice filters by thresholds and central tendency, as in content-validity ratios (VISION-Cog) and international priority-setting panels (Delphi consensus on behavioral trial methodology). An LLM-based Delphi Study to Predict GenAI Evolution automates the coordinator but deduplicates by embedding similarity, which groups by topic rather than by who agrees. Separately, Voting or Consensus? Decision-Making in Multi-Agent Debate documents "drift" in LLM debates: extra rounds lower accuracy through sycophancy, and consensus protocols reach 28.4% versus 54.8% for voting on the reasoning benchmark MuSR. If iteration with convergence-seeking feedback degrades machine panels, it is a live hypothesis that it distorts human ones. That result constrains us. We cap rounds at three and treat "more rounds" as a variable, not an assumption.
The human-centric lens comes from Deference by Design. It argues that when contesting a displayed output is costly, rational users accept it, and that pluralism therefore fails at the interface. That paper surfaces alternatives to one user and reports no human-subject numbers. We extend its concern to group feedback and give it a quantitative test. The causal framing is borrowed from Towards a Post-Market Monitoring Framework for Machine Learning-based Medical Devices. That paper shows that when predictions change the behavior they predict, observational metrics mislead and a counterfactual estimand is needed. The analogue here is that representation measured against round-$t$ preferences is circular when round-$t$ preferences were shaped by round $t-1$ feedback.
Research Program
The feedback engine. In each round, every panelist gives ratings on the items under consideration plus a short free-text rationale. The rationales are the candidate pool. An LLM discriminative oracle, in the style of Fish et al., predicts each panelist's approval of each rationale. A random 10% of these predictions are verified by asking the panelists directly, which lets us estimate oracle noise and report it against PROSE's noise parameter. The slate is selected by word-budgeted Method of Equal Shares, with sequential Phragmén as a secondary rule. The budget matches what the standard Delphi arm shows, so all arms impose the same reading load. Rationales are shown verbatim and anonymized. We deliberately use selection over existing rationales rather than LLM rewriting, so that positivity bias in generated text cannot enter. The generative variant is tested only as an ablation.
Outcome measures. Let $y_i^{(r)}$ be panelist $i$'s rating in round $r$, and let $\tilde y_i$ be a private re-rating taken one week after the panel. That re-rating is collected with no feedback shown and an explicit invitation to restate one's own view. Displayed convergence is the drop in panel dispersion from round 1 to round 3. Endorsed convergence is the same drop computed with $\tilde y_i$ in place of $y_i^{(3)}$. The deference gap is their difference.
Minority survival is defined using factions identified from round-1 ratings, before any feedback has been shown. It is the fraction of factions of at least $n/k$ members, with $n$ panel size and $k$ slate size, that are still represented in every round's feedback by a statement they approve.
Contestation rate is the share of panelists who submit a rationale that disagrees with the slate. Time-on-feedback is logged in the CoAuthor event-logging style (CoAuthor).
External representation follows Fish et al.: a fresh cohort that never participated rates how well the final recommendation and its stated rationale represent them. This addresses the circularity problem directly.
Causal design. The no-feedback arm, where panelists simply re-rate over three rounds, is the counterfactual. Movement in that arm reflects reflection and noise. Movement beyond it in each feedback arm is attributable to that arm's feedback.
Baselines
The naive reference is the no-feedback arm. Panelists repeat their ratings across three rounds and see nothing. Any feedback arm that cannot beat it on endorsed convergence or external representation is adding nothing but pressure.
The incumbent is standard Delphi feedback: median and interquartile range per item plus a coordinator's selection of rationales. To make it reproducible and fair, the "coordinator" is an LLM instructed in the usual Delphi way, highlighting representative and outlier justifications. It works under the same word budget. No reported result found on Lacuna for minority survival or deference under standard Delphi feedback, which is part of the gap.
The strongest existing method for the selection step is PROSE (Generative Social Choice: The Next Generation), run once per round on the same rationale pool with the same budget and oracle model. Its reported static advantage over clustering and zero-shot baselines is 10–40% mean satisfaction, 40% bottom-quartile utility, and 7–49 points fewer BJR violations. Its reported runtime is 31–65 minutes per run, which is acceptable for asynchronous rounds lasting days. Our extractive Equal Shares rule beats it if it matches PROSE's external representation and minority survival at a fraction of the oracle calls. If PROSE is clearly better, the generative step matters and we adopt it.
Two further baselines are top-$k$ approval selection, the "excellence" rule from the trichotomy of Faliszewski-style comparisons, and $k$-means representatives over sentence embeddings, which matches the LLM-Delphi study's similarity filtering. Both run on the same oracle predictions and budget.
The key ablation removes weight reduction. Plain approval over the same oracle and candidates differs from Equal Shares only in whether approving voters' weight is spent, so any difference isolates the proportional ingredient.
Beating the incumbent means two things together. Minority survival must be higher, measured on panels where round-1 factions exist. The deference gap must be smaller, with endorsed convergence not significantly below the incumbent. We pre-register a non-inferiority margin for that last condition.
Why not just…?
Why not just show every rationale? That is maximally representative and needs no algorithm. The trouble is that Delphi panels already suffer heavy fatigue and attrition (An LLM-based Delphi Study to Predict GenAI Evolution). Deference by Design argues that high reading and contesting costs push people toward accepting whatever is most salient. With 50 panelists and several items, "everything" means pages of text in which ordering effects silently decide what gets read. We include a full-display arm in the LLM-simulated pilot, and in one human arm on the smallest topic. If it matches proportional feedback on endorsed convergence without higher attrition, the selection step is unnecessary at that scale.
Why not just sample rationales at random? This is the most serious threat. A random sample gives every faction its proportional share in expectation, with no oracle and no algorithm. The proportional rule's advantage is a per-instance guarantee, which matters only when slates are small and factions are near the $n/k$ threshold. A 15% faction with $k=5$ will often get zero seats at random. We therefore include a random-$k$ arm, and we vary faction size in the simulated pilot from 10% to 40% to map where the guarantee pays off. If random sampling matches Equal Shares on minority survival at realistic panel sizes, we will say so and recommend it.
Why not just run PROSE once at the end and skip the iterative rounds? One-shot proportional summarization is already done and works (Generative Social Choice). But a Delphi panel's purpose is revision in light of others' reasoning, and that is exactly what produces the implementation adjustments seen in FootyFirst (nogaps). Comparing the round-1-only slate with the round-3 slate on external representation tells us directly whether iteration adds value. If it does not, that is a useful negative result for Delphi practice.
Why not just simulate the panel with LLM agents? LLM agents mirror demographic bias and lean on identity cues (To Mask or to Mirror), show positivity bias, and drift under debate. We use simulation only as a cheap pilot to tune budgets and faction sizes. We report how well simulated deference gaps predict human ones, as a secondary result rather than a replacement.
Proposed Experiments
The seed topics and initial statement pools come from three sources: the Bowling Green civic deliberation data and UCI Drug Review subsets used by PROSE, and the US abortion policy prompt from Fish et al. Using these lets the selection step be checked against published one-shot numbers before any iteration. A sanity stage reruns Equal Shares, sequential Phragmén and PROSE on these static pools. This confirms we reproduce the direction of PROSE's reported gains, and measures our oracle's verified noise.
The pilot stage runs LLM persona panels over the same topics, with planted factions of controlled size and seven feedback conditions, across one to five rounds. It maps where proportional selection diverges from random sampling and tests whether drift appears in a Delphi setting.
The main study recruits human panels of about 50 per arm across three topics. Recruitment is stratified so that at least one round-1 faction falls between 10% and 25%. Five arms run for three rounds: no feedback, standard Delphi, top-$k$ approval, random-$k$, and Equal Shares. A one-week private re-rating follows, and a fresh 100-person cohort gives external representation ratings.
The decisive checks are four. Equal Shares must raise minority survival over standard Delphi and top-$k$. It must shrink the deference gap relative to standard Delphi, with endorsed convergence within the pre-registered margin. Its advantage over random-$k$ must be concentrated where factions sit near the $n/k$ threshold, which is the specific prediction of the per-instance guarantee. Finally, the no-feedback arm must show less displayed convergence than any feedback arm; otherwise the causal attribution fails. If proportional feedback wins on survival but loses on endorsed convergence, then it trades agreement for representation rather than improving deliberation, and we will report it as such.
References
- Generative Social Choice
- Generative Social Choice: The Next Generation
- Phragmén's Voting Methods and Justified Representation
- Proportional Justified Representation
- An Experimental Comparison of Multiwinner Voting Rules on Approval Elections
- Feasibility, comprehensibility and acceptability of the VISION-Cog, a novel tool to assess cognitive impairment in visually impaired older adults: a cross-sectional pilot study in Singapore
- An international, Delphi consensus study to identify priorities for methodological research in behavioral trials in health research
- An LLM-based Delphi Study to Predict GenAI Evolution
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- Deference by Design: Pluralistic Alignment Is an Interface Problem
- Towards a Post-Market Monitoring Framework for Machine Learning-based Medical Devices: A case study
- CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
- Bridging the gap between research evidence and real-world implementation: the nogaps project
- To Mask or to Mirror: Human-AI Alignment in Collective Reasoning