The Delphi method is how medicine writes guidelines when trial evidence is thin. Experts rate statements anonymously. They see a summary of everyone's ratings, then re-rate over several rounds until a preset share agrees. The cartilage-repair panel of 75 experts reached "strong consensus" on 10 of 12 statements, yet most of those statements had to be graded as expert opinion only (Conservative Management and Biological Treatment Strategies). Nobody checks whether the rounds pulled the panel toward the truth or toward whatever most panelists happened to have read. This matters most when the evidence itself is heterogeneous. If half the trials define an outcome one way and half another, each expert absorbs a different slice of the literature. The feedback rounds can then either pool those slices or lock in the majority's slice.
We propose to treat a Delphi panel as a stochastic cellular automaton: a grid or graph of cells (here, experts) whose discrete states (their ordinal ratings) update each round by a simple local rule that depends on the feedback they are shown. We fit that rule to panels of language-model experts that each see a controlled, deliberately unequal share of real clinical-trial records. Every question has a known answer: whether the drug was later approved. We then use the fitted automaton to map when consensus tracks evidence and when it locks in, and to design feedback protocols that stay on the evidence-tracking side.
Core Claim
Our claim has two parts. First, round-to-round rating changes in a Delphi panel are well described by a local stochastic rule with three forces: pull toward the feedback statistic the expert is shown, pull toward what the expert's own evidence implies, and a constant bias. Concretely, for expert $i$ moving from round $t$ to $t+1$:
$$ P(r_i^{t+1} = r) \propto \exp\big(-\alpha\,|r - m_{N(i)}^{t}| \;-\; \beta\,|r - e_i| \;+\; h\,r\big) $$
Here $r$ is a rating on the panel's ordinal scale, $m_{N(i)}^{t}$ is the median rating over the set $N(i)$ of panelists whose ratings and rationales expert $i$ is shown, and $e_i$ is the rating implied by the studies expert $i$ was given. The parameter $\alpha$ is conformity pressure, $\beta$ is evidence anchoring, and $h$ is a uniform field that captures the documented positivity bias of language-model panelists (An LLM-based Delphi Study to Predict GenAI Evolution). The rule has the same Metropolis-style form that drives the phase transitions of the separation algorithm of A Local Stochastic Algorithm for Separation in Heterogeneous Self-Organizing Particle Systems. That paper proves two-colour particles on a lattice separate once affinity parameters cross a threshold. Our "colours" are evidence exposures, and the analogous question is whether the panel separates by exposure or merges onto the evidence.
Second, we predict that panel outcomes show a sharp change, not a gradual drift, as the ratio $\alpha/\beta$ rises against the imbalance in evidence exposure. Above the threshold, the final consensus follows majority exposure regardless of which evidence is more predictive. Standard Delphi uses global median feedback, where $N(i)$ is the whole panel, and we expect it to sit on the wrong side of this threshold under realistic heterogeneity. Restricting $N(i)$ to a structured local neighbourhood, so that each expert sees rationales from a chosen subset, should preserve minority evidence longer. This mirrors the traffic result in which local routing agents stayed stable at scale where a global optimizer degraded (Comparison between Global and Local Optimization Routing Method for Alleviation of Traffic Congestion).
Where this sits
Three lines of prior work each cover part of this space.
Delphi–heterogeneity work uses consensus to standardize definitions but never checks the consensus against outcomes. The Vascular Impairment of Cognition Classification Consensus Study unified dementia nomenclature with a 67% cutoff. The cartilage panel set treatment pathways. Both motivate our question, and neither has ground truth.
Automated Delphi supplies our testbed method. An LLM-based Delphi Study to Predict GenAI Evolution ran five-round panels of persona-prompted agents with an organizing agent, and reported thematic consistency. It measured consistency, not correctness, and its agents saw no data. We keep its two-agent architecture but give each responder a defined evidence packet, and we score answers against outcomes. AI Agents and the Future of Deliberation warns that ungrounded AI deliberation threatens validity. That warning constrains us: we make no claim that agent panels replace humans, only that they are a controllable instrument for studying protocol dynamics.
Cellular-automaton work supplies the modelling and tooling. The separation algorithm gives the rule family and the phase-transition lens. Learning Graph Cellular Automata lets us learn a non-parametric update rule on an arbitrary graph, which serves as a flexible competitor to our three-parameter rule. CAX makes it cheap to sweep millions of simulated panels. Prior CA work studies synthetic binary heterogeneity. Here, heterogeneity comes from real trial records.
Research Program
Testbed. We build the testbed from the TOP trial-outcome benchmark used by HINT: 7,062 trials with approval labels. Each Delphi question asks whether a drug–indication pair will succeed in a given phase. Evidence packets are earlier trial records and linked metadata for that drug or indication. We group records by outcome definition using the 200 cluster-outcome nodes of CTKG, which turns real outcome-definition heterogeneity into an explicit label. We then control how packets are split across panelists. Splits range from "everyone sees everything" to "70% of panelists see only definition cluster A, 30% only cluster B." The more predictive cluster is identified in advance on training questions.
Data. We run agent panels across panel sizes, round counts, consensus cutoffs (51%, 67%, 75%), and feedback types. The feedback types are median only, median plus all rationales, and median plus rationales from a structured neighbourhood. We log every rating and rationale per round.
Model fitting. We fit the three-parameter rule by maximum likelihood on round-to-round transitions. We also fit a learned graph cellular automaton and a linear opinion-averaging model as competitors.
Analysis. In CAX we sweep $\alpha$, $\beta$, $h$, exposure imbalance and neighbourhood structure to produce a phase diagram. We locate the lock-in boundary and compare it with a mean-field approximation of the rule.
Protocol design. We use the fitted model to choose a protocol: feedback type, neighbourhood, number of rounds and cutoff. The model predicts this protocol maximizes evidence tracking, and we validate it prospectively on held-out questions.
As a qualitative human check, we extract the round-by-round agreement percentages that published Delphi reports such as VICCCS and the cartilage panel give. We ask whether their trajectories fall in the fitted model's lock-in or tracking region. We treat this as illustration, not validation.
Baselines
Naive reference: round-one median. This is the panel's independent ratings with no iteration, the pure wisdom-of-crowd answer, run on identical evidence splits. If Delphi rounds do not beat it, iteration adds nothing. A designed protocol must beat it on PR-AUC (area under the precision–recall curve) and Brier score with a paired bootstrap confidence interval excluding zero.
Second naive reference: one model with all evidence. A single language model given all packets at once tests whether the panel structure matters at all.
Strongest existing protocol baseline. This is the Bertolotti–Mari automated Delphi with global median feedback and organizing-agent reformulation. It is reimplemented on the same base model, the same packets, the same number of model calls, and the same prompt-tuning budget on training questions.
Strongest existing predictor. This is HINT, trained on TOP. HINT reports PR-AUC of 0.772 in Phase I, 0.607 in Phase II and 0.623 in Phase III (HINT). LIFTED reports 71.6% F1 in Phase II. We do not expect a panel to beat a supervised model trained on thousands of labelled trials. HINT is the ceiling that tells us how much predictive signal the evidence carries. We report each panel's accuracy as a fraction of HINT's on the same held-out split.
Modelling ablations. These isolate the key ingredient, the local stochastic rule with an evidence anchor:
- linear opinion averaging, in which each expert moves toward a weighted mean;
- our rule with $\beta=0$, which removes the evidence anchor;
- our rule restricted to global neighbourhoods;
- the learned graph automaton.
The automaton earns its place only if it beats linear averaging on held-out transition log-likelihood. It must also predict which option wins and the final agreement level on held-out panel configurations, especially near the predicted boundary.
Why not just…?
Why not just run more agent panels and read off the best protocol empirically? For the conditions we sample, direct search would work. That is why we run it as the validation arm. Its weakness is cost and coverage. Real panels vary in size, cutoff, rounds and exposure structure combinatorially, and agent runs are expensive. A cheap, fitted model can extrapolate to unsampled regimes such as 200-expert panels only if it predicts held-out configurations. Experiment two tests exactly that. If the model fails to predict unsampled configurations, we fall back to direct search and the automaton claim fails.
Why not just use a classic linear consensus model? This is the most serious threat. Repeated weighted averaging with stubbornness is simpler and well understood. It cannot, however, produce discontinuous outcomes from discrete ratings and hard consensus cutoffs, which are the features that make Delphi lock-in possible. If linear averaging fits transitions and final outcomes as well as our rule, we will report that and frame the contribution around the phase diagram it implies, not around automata.
Why not just pool the trials with random-effects meta-analysis and skip the panel? When outcome definitions conflict, pooling often has to be abandoned for narrative summaries (Clinical complete response as a surrogate for pathological response in bladder cancer is one example of definitional variability breaking a surrogate). Expert panels exist precisely for these cases. We nonetheless include one feedback variant that shows panelists a pooled estimate wherever pooling is possible. If that variant dominates, the practical lesson is "ground Delphi feedback in pooled data." That would be a useful result in its own right, and it is something the model can quantify.
Proposed Experiments
Experiment one: does lock-in exist? We hold the evidence fixed and vary exposure imbalance and feedback type. The decisive check is whether, under global median feedback, the probability that the final consensus follows the majority-exposure cluster jumps sharply past an imbalance threshold. The jump must occur even when the minority cluster is the more predictive one. A smooth, linear dependence would falsify the phase-transition claim.
Experiment two: does the automaton predict held-out panels? We fit on one set of configurations and predict final consensus, its direction and the number of rounds to cutoff on configurations never run. We compare the three-parameter rule, the learned graph automaton and linear averaging. Predictions are scored by log-likelihood and outcome accuracy.
Experiment three: does model-designed feedback improve accuracy? On held-out TOP questions, we compare the protocol chosen by the fitted model against standard global-feedback Delphi and the round-one median. We report PR-AUC by phase alongside HINT's reported values, Brier score, and a lock-in index: the fraction of final-answer variance explained by exposure share after controlling for evidence content.
Experiment four: does the positivity field transfer? We repeat Experiment three with a second base model. The test is whether $h$ changes while $\alpha$ and $\beta$ stay stable. Stability would indicate the rule captures protocol dynamics, not one model's quirks.
The whole programme stands or falls on Experiments one and two. If panels show no sharp lock-in, or a linear model predicts them as well, the automaton view adds nothing and we say so.
References
- Conservative Management and Biological Treatment Strategies: Proceedings of the International Consensus Meeting on Cartilage Repair of the Ankle
- An LLM-based Delphi Study to Predict GenAI Evolution
- A Local Stochastic Algorithm for Separation in Heterogeneous Self-Organizing Particle Systems
- Comparison between Global and Local Optimization Routing Method for Alleviation of Traffic Congestion
- The Vascular Impairment of Cognition Classification Consensus Study
- AI Agents and the Future of Deliberation: Designing Human-AI Collaboration for Democratic Dialogue
- Learning Graph Cellular Automata
- CAX: Cellular Automata Accelerated in JAX
- HINT: Hierarchical interaction network for clinical-trial-outcome predictions
- A knowledge graph of clinical trials ($$\mathop {\mathtt {CTKG}}\limits$$)
- LIFTED: Multimodal Mixture-of-Experts for Clinical Trial Outcome Prediction
- Clinical complete response as a surrogate for pathological response in bladder cancer: a systematic review and meta‐analysis