The Delphi method is a structured way to get a panel of experts to agree. Experts rate items anonymously, see the group's median and the anonymous reasons behind it, revise their ratings, and repeat until the ratings stop changing. It underpins clinical guidelines, questionnaire design, and even the field's own evaluation standards. Yet Delphi has an awkward property: its outputs are almost never checked against the truth. The method is used precisely where ground truth is unavailable, so studies report how much the panel agreed rather than whether the panel was right. Recent LLM-augmented versions report impressive agreement numbers, but agreement is not accuracy, and the iteration that drives agreement could just as easily drive herding.
We propose to test Delphi where the truth can be computed: small sequential games. In these games, a single solver configured with a few hyperparameters can be run exhaustively to find the best configuration for each game. Expert panels, human and LLM, will predict which solver configuration (which distance function in the update, how many past policies to mix, which step size) works best on games they have never seen run. We then score those predictions against exhaustive search. This gives the first controlled measurement of whether Delphi's three ingredients (anonymity, iteration, controlled feedback) make a panel more correct, or only more unanimous. As a by-product, it yields an expert "fingerprint" that maps game properties to solver configurations, which can warm-start the solver's own tuning.
Core Claim
Exactly solvable decision problems are a missing validation ground for consensus methods. Delphi's recommendations, and the confidence tiers that recent hybrid variants attach to them, can be scored there by regret against an oracle. Our central hypothesis is directional but falsifiable: iteration with controlled feedback lowers a panel's regret relative to its own first-round median. A competing hypothesis is equally publishable: iteration raises agreement while leaving regret flat or worse. Either outcome changes how Delphi results elsewhere should be read.
The three directions play different roles here. The unified decision-making line supplies the testbed and the ground truth. Configurable Mirror Descent (CMD) is a single solver for single-agent, cooperative, competitive, and mixed-motive games. It updates policies by mirror descent, meaning gradient steps measured in a chosen distance function called a Bregman divergence. A meta-controller tunes its hyperparameters by zero-order random search. The paper ships GAMEBENCH, 15 small games, and reports that the best divergence depends on the game: squared Euclidean beats KL on Kuhn Poker. CMD explicitly leaves divergence selection to manual choice, which makes "which configuration fits this game?" a natural question to put to experts.
The Delphi line supplies the method under test and the claims we want to check. The Human-AI Hybrid Delphi Model pairs Gemini 2.5 Pro with six-expert panels. It reports 95% replication of historical consensus items, 95% directional agreement with sleep specialists, and 92.3% single-round consensus, and it proposes a four-tier consensus label running from Strong to Divergent. Metrics reloaded used 73 experts reaching 93% consensus to build "problem fingerprinting," a flowchart mapping task properties to evaluation metrics. Every one of these figures measures agreement among people or between model and people. None measures correctness against an external answer.
The empirical-validation line supplies the constraint. Trust Models and Con-Man Agents showed that trust models that are provably forgiving are empirically exploitable by agents who alternate cooperation and defection. That warning applies to any feedback loop that forgives. We take it as the template: theoretical or social guarantees must be stress-tested in simulation, and we do the same to Delphi's guarantee of convergence to collective judgment.
What is new is not putting Delphi inside a solver. It is using solvers to put Delphi under measurement.
Research Program
Ground truth. For every game we run CMD over a full grid of configurations. The grid covers the divergence family (KL, squared Euclidean, polynomial), the number of historical policies mixed, and the step size. We record the final value of the game's own objective: optimality gap for single-agent games, NashConv (how much any player could gain by deviating) for competitive games, and social welfare for cooperative and mixed games.
Games. We start from the 15 GAMEBENCH games. We add parametric families that can still be solved exactly: Kuhn and Leduc variants with different deck sizes and bet structures, random matrix games with controlled zero-sum versus general-sum structure, and small cooperative instances built with the MADP Toolbox. This yields a few hundred games, split into an "evidence" set and a held-out "prediction" set.
Panels. Panelists see full results on the evidence set, playing the role of the literature in a clinical Delphi. On held-out games they see only rules and payoff structure. For each game they rank configurations, give a confidence, and write a short rationale. Human panels are six researchers in game theory or reinforcement learning, a size taken from the saturation finding of HAH-Delphi. We run several independent panels so that we can estimate panel-to-panel variance. LLM panels are many replicate groups of model instances, which lets us ablate protocol components with statistical power that human panels cannot give.
Protocol arms. The full Delphi arm uses anonymous ratings, a median and interquartile-range summary, anonymized rationales, and rounds that continue until rankings are stable. One arm drops anonymity, so names or personas are visible. One arm drops iteration and simply aggregates the round-one ratings. One arm drops the rationales and feeds back numbers only. One arm is unstructured free chat, the communication mode whose failures Collab-Overcooked documents. Each final ranking is labeled with the HAH-Delphi four-tier consensus category so that we can test whether the tiers track correctness.
Fingerprint. We distill the pooled panel judgments into a small decision chart in the style of Metrics Reloaded, mapping game properties to recommended configurations. We then evaluate the chart as a warm start for CMD's random-search meta-controller.
Baselines
Default configuration. The naive reference is CMD's KL divergence with default settings, which corresponds to the MMD-KL style setting the CMD paper compares against. A uniformly random configuration serves as a floor.
Oracle. Exhaustive grid search defines zero regret. Every arm is scored by normalized regret: the gap between the chosen configuration's final objective and the oracle's, divided by the gap between the default and the oracle.
Per-game meta-controller search. The strongest competitor for the configuration task is CMD's own meta-controller run separately on each game with a budget of $k$ evaluations. It is strongest because it observes actual outcomes rather than guessing from rules. The CMD paper reports convergence on all 15 games and lower error than MMD-KL in many of them. However, the exact table values were not transcribed, so there is no reported result on Lacuna for per-game NashConv or optimality gap under CMD. We therefore regenerate these numbers ourselves with the released settings. The fingerprint "beats" this search if warm-starting from it reaches within a fixed fraction of oracle regret using fewer evaluations than cold-start search, on held-out games and with an identical total evaluation budget.
Learned selector. A small classifier is trained on the evidence games. It uses structural features (number of players, zero-sum flag, information-set count, chance nodes) to predict the best configuration. This is the data-driven competitor to expert judgment and is trained on exactly the information panelists see.
Iteration and anonymity ablations. These isolate the proposal's key ingredient. Round-one aggregation, whether by mean or majority, is the "wisdom of crowds without Delphi" reference. Delphi beats it only if final-round regret is lower in a paired comparison across held-out games. Agreement gains alone do not count.
Why not just…?
Why not just run the meta-controller on every game and skip experts entirely? On tabular games, search is cheap, and for the narrow task of configuring CMD this is a real threat. We concede it directly: the warm-start result is secondary. The primary contribution is measuring Delphi, and for that question the solver is the ruler, not the competitor. The warm-start experiment will tell us honestly whether expert priors save any search. If they do not, we report that.
Why not just learn the configuration map from game features? With a few hundred games, a learned selector may match or beat the panels. That would itself be informative, because it would mean structured expert consensus adds nothing over a small classifier fed the same evidence. We compare the two head to head on identical held-out games. We also test their combination, using panel rankings as a feature, to see whether experts carry information that the features miss.
Why not just average independent judgments and drop the rounds? This is the core skeptical alternative, and it is exactly what the no-iteration arm tests. HAH-Delphi argues that averaging causes "interpretive flattening," the loss of conditional "it depends" caveats. It does not show that iteration improves correctness. Our paired-regret design settles the question in either direction.
Why not just let LLM agents discuss freely? Collab-Overcooked reports that 13 LLMs collapse at difficulty level 4 and above in open collaboration. It also reports that manually redirecting their attention improved performance by 35% to 64%. That evidence suggests structure matters, but it comes from a different task. The free-chat arm measures the effect directly here.
Proposed Experiments and Decisive Checks
Accuracy against truth. The primary measurements are normalized regret, top-1 accuracy, and Kendall rank correlation with the oracle ranking on held-out games. Each is reported per protocol arm and separately for human and LLM panels. We run paired tests across games, and for LLM panels we use bootstrap over panel replicates.
Do iteration and confidence tiers mean anything? Two analyses decide what Delphi is worth. First, we check whether final-round regret falls below round-one regret within the same panels. Second, we check whether items labeled Strong Consensus have measurably lower regret than items labeled Conditional or Divergent. If agreement rises while regret does not fall, we will have documented herding. That would directly qualify the agreement-only validations in HAH-Delphi and Metrics Reloaded.
Robustness to a manipulative panelist. Following the con-man analysis, one LLM panelist in each panel is instructed to argue persistently for a fixed wrong configuration. We measure how far it shifts the panel's final regret. We also test whether anonymized rationale feedback dampens or amplifies that shift compared with numbers-only feedback.
Generalization of the fingerprint. The distilled fingerprint is tested as a CMD warm start on the held-out games, including the cooperative MADP instances that are structurally furthest from the poker variants. This shows whether expert-derived rules transfer across game classes or only within the families panelists studied.
References
- Configurable Mirror Descent: Towards a Unification of Decision Making
- The Human-AI Hybrid Delphi Model: A Structured Framework for Context-Rich, Expert Consensus in Complex Domains
- Metrics reloaded: recommendations for image analysis validation
- Trust Models and Con-Man Agents: From Mathematical to Empirical Analysis
- The MADP Toolbox: An Open Source Library for Planning and Learning in (Multi-)Agent Systems
- Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agents