Expert consensus panels are used when hard evidence is missing: clinical guidelines, organ allocation rules, pandemic responses. The standard tool is the Delphi method, in which experts answer a questionnaire anonymously, see the group's statistics and reasons, and answer again over several rounds. Delphi has a known weakness. Experts often disagree because each carries a different unstated assumption ("closing schools works if transmission is already concentrated in children"). Aggregating their ratings into a median discards exactly that conditional structure. Meanwhile, many of these domains already have simulators that could test such assumptions. These are agent-based epidemic models, transit digital twins, and organ-allocation sandboxes, built as libraries that can be paused, inspected, and changed mid-run.
We propose a Delphi protocol in which a panelist's justification is a runnable claim rather than a paragraph of text. Between rounds, each dissenting panelist's conditional claim is compiled into a counterfactual branch of a shared simulation, started from a common checkpoint. The panel's anonymous feedback then shows both the rating distribution and how each stated condition actually played out. The bet is that this separates two kinds of disagreement that classic Delphi blends together. Empirical disagreement, about what would happen, should shrink, and shrink toward the truth. Value disagreement, about what outcome is worth more, should survive and be labelled as such rather than flattened into a false consensus.
Core Claim
Panelist-authored counterfactual branches, run from shared simulation checkpoints and returned under Delphi anonymity, will make consensus estimates more accurate than classic or literature-augmented Delphi on questions with a knowable answer. They will do so without increasing false convergence on questions that hinge on values. The ingredient we claim matters is that the panelists, not the coordinator, choose which conditions get simulated. Low-latency steering is a means, and we test whether it matters rather than assume it.
As far as the research file shows, nobody has joined these two lines of work. The closest work is The Human-AI Hybrid Delphi Model, which pairs a language model acting as an "evidence floor" with a small human panel. It introduces a four-tier outcome taxonomy (strong, conditional, operational, no consensus). It reports 95% replication of historical consensus items, 95% directional agreement with senior experts, and consensus on 92.3% of items in a single round. Its evidence is static literature, and its "conditional consensus" tier records conditions without testing them. We reuse its taxonomy as our outcome labelling scheme. Where its conditions stay recorded caveats, ours become executable branches.
On the simulation side, High performance informatics for pandemic preparedness and Scenario-based Digital Twin Evaluation for Visualizing Bus Planning Policy Measures both expose "what-if" interfaces to decision makers. In both cases decisions are reached through ad hoc analyst queries or a single user moving sliders, with no protocol for structured group disagreement. These papers motivate our setting but supply no consensus mechanism.
Research Program
The work has three parts: a branching substrate, a claim-to-branch translation step, and the protocol itself.
The branching substrate. This is a thin layer over simulators that are already exposed as libraries. PyPartMC wraps a Fortran aerosol code so that Python can hold its state arrays and drive its time loop, at roughly 1% overhead for 10,000 particles. SpeedyWeather.jl makes an atmospheric model inspectable mid-run. Both show that state can be held, copied, and modified from a scripting layer, and we reuse that pattern for checkpoint-and-fork. CausalWorld supplies the vocabulary for interventions: it treats the simulator as a structural causal model with a programmatic interface for changing variables mid-episode. Every panelist claim will be expressed as an intervention of the form "at time $t$, set variable $v$ to value $x$, then report outcome $y$."
Work on decoupled in-situ rendering (Gupta et al.; Sbalzarini) tells us not to render per panelist on the compute path. Branch outputs are reduced to small summary trajectories before anyone sees them.
Claim-to-branch translation. Panelists write justifications in a structured form: condition, intervention, and expected direction of effect. A language-model translator turns the form into intervention code, and the panelist confirms it before it runs. LLMR shows that text-to-code injection into live scenes works but is error-prone. It reports error rates falling from about 66% to 14% in empty scenes, and from 85% to 21% in complex ones, with an inspector agent. Those error rates are why the panelist confirmation step is mandatory, and why we log translation errors as a separate outcome.
The protocol. It keeps Delphi's three pillars: anonymity, iteration, and controlled feedback. It changes what the feedback contains. After round $k$, every claim behind an out-of-band rating is run as a branch, alongside the modal panel view, from the same checkpoint and with the same random seeds. Round $k+1$ feedback shows three things: the rating distribution, the anonymized claims, and each claim's simulated outcome with seed variability.
The motivation for using trajectories rather than endpoints comes from Toward Rigorous Parameterization of Underconstrained Neural Network Models. In that work, steering a whole-brain model showed that different parameter paths reaching the same final activity produced structurally different, sometimes non-viable networks. So branches report the full outcome trajectory, not just its final value.
Each item finally receives a label from the four-tier taxonomy. We add one tag: whether the remaining dispersion is explained by differing outcome weights, which panelists state explicitly, or by differing outcome predictions.
Baselines
Classic modified Delphi. This is the do-nothing reference: anonymous rounds with medians, interquartile ranges, and text justifications, as used in the behavioral-trials prioritization Delphi and the ankle cartilage consensus meeting. It runs on identical items and the same number of rounds, with panelists drawn from the same pool. It is beaten if our consensus estimates are more accurate on ground-truth items without being more falsely convergent on value items. It is also the reference for attrition, where the behavioral-trials study lost 20 of 77 panelists between rounds.
HAH-Delphi. This is the strongest existing method (The Human-AI Hybrid Delphi Model). It is strongest because it already attacks interpretive flattening and has the best reported convergence: 92.3% of items in a single round, and 95% directional agreement with experts. We run it with the same panel size, the same language model access for evidence synthesis, and the same facilitator time. Matching its convergence rate is not the target, since fast convergence is cheap if it is wrong. It counts as beaten only if our protocol is more accurate on ground-truth items at comparable convergence.
Coordinator-designed offline sweep. This is the key ablation. The coordinator reads round-1 justifications and pre-computes a parameter sweep with the same total simulation compute as our branches. The results are circulated as static figures. It isolates whether panelists authoring the counterfactuals matters, as opposed to merely having simulation evidence. If it matches our protocol, the claimed ingredient is wrong.
Interactive versus overnight execution. This is a second ablation. Panelists either steer branches themselves within a round, or submit claims that run as overnight batch jobs. It tests whether live steering adds anything over on-demand batch execution.
Synchronous screen-share workshop. A third comparison, in which an operator drives the simulator while experts discuss openly. It measures what anonymity buys once everyone sees the same simulations.
Why not just…?
Why not just pre-compute a parameter sweep and email the charts? This is the most serious threat, and we treat it as a likely competitor rather than a strawman. For many questions the conditions that matter are guessable in advance, and a coordinator's sweep would cover them. Our argument is narrower. Panelists' conditions are often state-dependent: "if hospital load exceeds a threshold before day 30." They are also idiosyncratic, so a coordinator's grid, at equal compute, spends most runs on conditions no one holds and misses the ones that drive disagreement. The coordinator-sweep ablation settles this directly. If its accuracy and its rate of resolved empirical disagreements are within noise of ours, we will report that the interface, not the protocol, was the contribution, and recommend the cheaper sweep.
Why not let a language model or an optimizer produce the recommendation? For the empirical part of the question, an optimizer over the simulator can do well, and we include a simulator-only answer as a reference. But the simulator is not the world, and the questions mix prediction with values. HAH-Delphi's authors found the model could not supply experiential or ethically grounded justification, so it served only as an evidence floor. CausalWorld shows that model-free learners fail under strong interventions; none solved two-block stacking. That result speaks to agents learning inside a simulator, so it is weak evidence about recommendation quality, and we do not lean on it. The decisive test is the misspecified-simulator condition below. There, a simulator-only answer is wrong by construction, and the question is whether the human panel catches the misspecification.
Why not run a live workshop around the digital twin? It is faster and needs no new protocol. It also brings back dominance and status effects, which Delphi's anonymity exists to remove. The workshop baseline measures this directly. If the workshop is as accurate as our protocol, and its outcomes are not predicted by panelist seniority, anonymity is not earning its cost here.
Proposed Experiments
Study 1: controlled ground truth. Delphi outcomes usually have no ground truth, so we build one. A held-out "reference world" is an agent-based epidemic model of a synthetic population, in the style of the CIEPI engines, with parameters withheld from panelists. Panels of 8 to 12 trained epidemiology participants receive a steerable model whose uncertain parameters span the reference values.
Items are of three types. The first asks for outcome estimates under timed interventions, such as peak hospital load under school closure at day 20 versus day 40; these have reference-world answers. The second asks for policy rankings. The third asks for explicitly value-laden trade-offs, where correct behavior is to remain "no consensus/divergent" or "conditional." Panels are randomized to the five conditions above.
The metrics are:
- error of the consensus estimate against the reference world, and calibration of the reported intervals;
- the fraction of round-1 disagreements resolved, split by whether they were empirical or value disagreements, with the split coded blind using the HAH-Delphi justification categories;
- false-convergence rate on value items;
- translation error rate;
- attrition across rounds.
Study 2: misspecified simulator. This is the decisive check. A simulator can manufacture confident wrong consensus. We therefore run a condition in which the steerable model is systematically biased relative to the reference world, for example by omitting a transmission setting. We compare its accuracy and confidence to classic Delphi. If executable Delphi converges more tightly on the wrong answer than classic Delphi, the method is unsafe as proposed. We would then redesign the feedback so it surfaces where model output and expert priors diverge, and test whether that remedy works.
Study 3: field deployment on organ allocation. We use AllSim on its UNOS liver data. AllSim already simulates supply shocks, such as a 50% drop in donor arrivals, and compares MELD-based against first-come allocation. No numerical score for those comparisons is reported on Lacuna, so we will establish our own simulator reference values before panels begin.
Transplant clinicians deliberate on allocation adjustments under a supply shock. Here there is no ground truth. We instead measure:
- stability across rounds;
- the share of items ending as conditional consensus with an executable condition attached, against HAH-Delphi on the same items;
- attrition, against the 77-to-57 loss reported in the behavioral-trials Delphi;
- panelist-reported burden per round.
There is no reported result found on Lacuna for any simulation-grounded Delphi, so Study 1's classic and HAH-Delphi arms serve as the reference points.
The proposal succeeds if two conditions hold together. First, panelist-authored branches beat the equal-compute coordinator sweep on accuracy for the empirical items. Second, they do not raise false convergence on value items or in the misspecified condition. If only accuracy improves, we have built a better persuasion tool, not a better consensus method, and we will say so.
References
- The Human-AI Hybrid Delphi Model: A Structured Framework for Context-Rich, Expert Consensus in Complex Domains
- High performance informatics for pandemic preparedness
- Scenario-based Digital Twin Evaluation for Visualizing Bus Planning Policy Measures
- PyPartMC: A Pythonic interface to a particle-resolved, Monte Carlo aerosol simulation framework
- SpeedyWeather.jl: Reinventing atmospheric generalcirculation models towards interactivity and extensibility
- CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
- A Proposed Framework for Interactive Virtual Reality In Situ Visualization of Parallel Numerical Simulations
- OpenFPM: A scalable open framework for particle and particle-mesh codes on parallel computers
- LLMR: Real-time Prompting of Interactive Worlds using Large Language Models
- Toward Rigorous Parameterization of Underconstrained Neural Network Models Through Interactive Visualization and Steering of Connectivity Generation
- An international, Delphi consensus study to identify priorities for methodological research in behavioral trials in health research
- Conservative Management and Biological Treatment Strategies: Proceedings of the International Consensus Meeting on Cartilage Repair of the Ankle
- AllSim: Simulating and Benchmarking Resource Allocation Policies in Multi-User Systems