Delphi Panels as Counterexample Oracles for Conditional Consensus Rules

A Delphi study is a structured way to get a group of experts to agree without meeting face to face. Experts rate statements anonymously, see a summary of how everyone else rated them, and re-rate over several rounds. It is how many clinical guidelines get written when trial evidence is thin. Its known weakness is that it aggregates ratings of pre-written statements. The conditional structure experts actually hold, such as "do X, unless the patient is frail and on anticoagulants," is either flattened into a median or left to a skilled facilitator who reads the free-text comments by hand. Exact learning, a branch of learning theory started by Angluin, studies the opposite setup. A learner reconstructs a precise rule by questioning a teacher. It can ask whether a specific case is positive (a membership query), or it can propose a whole rule and receive a case the rule gets wrong (an equivalence query answered with a counterexample). Those two query types turn problems that are hopeless from passive data into ones solvable with few questions.

This note proposes running a Delphi study as an exact-learning dialogue. The object being learned is a small, explicit decision rule, written as a shallow decision tree over case features. Each round shows the panel the current draft rule and asks each expert for one concrete case where they would decide differently. That case is then put back to the whole panel as a rating question, which decides whether it is a genuine panel-level counterexample or one expert's idiosyncrasy. Cases where the panel stays split are not averaged away. They become an explicit "no consensus" region of the rule. The expected payoff is a guideline that keeps its "it depends" conditions, for fewer expert responses than a large static vignette survey. We test that expectation against the natural baselines at equal expert-response budgets.

Core Claim

The claim is that the panel's consensus concept can be learned directly, at lower expert cost than passive elicitation, while explicitly marking where consensus fails. The panel's consensus concept is a three-valued function over cases: consensus-yes, consensus-no, and divergent, with the verdict set by a fixed supermajority threshold. The learning uses a counterexample-then-verify protocol. The part that is actually new is how the panel serves as the oracle. Exact-learning algorithms assume one noiseless teacher. A Delphi panel is several noisy teachers who disagree. The proposal splits these roles. Individual experts are cheap, untrusted sources of candidate counterexamples, which covers the equivalence-query role. The panel as a whole is the trusted but expensive judge that labels each candidate, which covers the membership-query role.

This design sits between two lines of existing work and is distinct from both. The Human-AI Hybrid Delphi Model adds a large language model to compact expert panels and reports 92.3% single-round consensus in an applied deployment. It also reports 95% directional agreement between the model and senior sleep experts. Its output is still ratings of fixed statements, synthesized manually through a seven-category coding of justifications. It notes that the model cannot supply the experiential "it depends" nuance. That study motivates the gap, but it has no query structure and no learned rule. Actively Learning Concepts and Conjunctive Queries under ELr-Ontologies learns ontology-grounded queries from a domain expert through membership and equivalence queries. It uses a refine step that minimizes and splits counterexamples. We reuse the idea of minimizing counterexamples, but that work models the expert as a single infallible oracle. Exact Learning of Preference Structure shows that a single-peaked preference axis over $m$ candidates is recovered in exactly $m-1$ equivalence queries. The authors assume noiseless voters who share one axis. That result gives us a clean noiseless reference point, and the question becomes how the result degrades with a disagreeing panel.

We are careful about what the theory does and does not give us. Exponential Separation between Two Learning Models and Adversarial Robustness proves that learning from counterexamples needs only $O(d\cdot\text{polylog}(1/\epsilon))$ queries. Here $d$ is the capacity of the rule class, measured by its VC dimension, and $\epsilon$ is the target error. Passive sampling instead needs $\Omega(1/\epsilon)$. That proof assumes realizability, meaning a perfect rule exists in the class, and an adversarial teacher. Neither assumption holds for human panels. We treat the separation as a hypothesis to test empirically, not a guarantee we inherit.

Research Program

The first component is a formal protocol. Let a case $x$ be a vector of clinically meaningful features. Let the panel have $k$ experts, and let $v(x)$ be the fraction of experts who endorse action A for case $x$. With a supermajority threshold $\tau$ (for example 0.7), the target is consensus-yes if $v(x)\ge\tau$, consensus-no if $v(x)\le 1-\tau$, and divergent otherwise. The hypothesis is a depth-bounded decision tree whose leaves may carry any of the three labels.

Each round has three steps. First, every expert sees the current tree and submits at most one case where they disagree with it. Second, a minimization step shrinks each candidate to its smallest disagreeing form, meaning the fewest non-default feature values that still flip that expert's judgment. This adapts the refine minimization of the ontology-learning work. It uses a few single-expert follow-up questions, or optionally an LLM draft that the expert confirms. Third, the minimized cases are batched and rated by the whole panel. The ratings update the tree through the membership-query-driven tree construction of Extracting PAC Decision Trees from Black Box Binary Classifiers, which extracts trees from a black-box model through a membership oracle. Here the panel replaces that black box.

The first step keeps the burden of generating counterexamples spread across experts. The third step is the only place the panel's collective verdict is spent. This is also where Delphi's anonymity and controlled feedback enter. Panel ratings of each counterexample are shown back only as anonymous summaries, and experts may revise their rating in the next round. That revision is exactly the kind of drift the exact-learning literature does not model.

The second component is modest theory for this oracle. Model each expert's response as the panel concept corrupted by independent flips with rate $\eta$. In that model a supermajority verdict on a batch costs $O(\log(1/\delta)/(\tau-1/2-\eta)^2)$ ratings per case, following standard concentration arguments, so that the verdict is correct with probability at least $1-\delta$. The divergent region is the set of cases whose endorsement rate lies near the threshold. It is identified only to within that margin. We will state query bounds for trees under this model and, more importantly, measure where they break.

The third component is empirical: simulated panels, the single-peaked testbed, and one real clinical Delphi study, all described below.

Baselines

The naive reference point is the classical static Delphi. A fixed battery of statements is written in advance, rated for two rounds with median and interquartile-range feedback, and retained by a content-validity threshold as in Follow-up after major traumatic injury. Its output is a list of endorsed statements. We convert that list into a rule for scoring by treating each retained statement as a leaf.

The strongest competitor is a passive vignette survey followed by tree fitting. Every expert rates a large random sample of synthetic cases. The supermajority label per case is computed, and a depth-bounded three-valued tree is fit to those labels. This baseline is dangerous because it uses the same rule class, the same threshold and the same expert pool. It differs only in who chooses the cases, so it isolates exactly the claimed advantage of letting experts supply counterexamples. There is no reported result found on Lacuna for vignette-survey tree fitting on consensus elicitation.

The third baseline is pool-based active learning. The learner draws candidate cases from a fixed unlabeled pool and sends to the panel those where the current tree's leaf is least certain.

The fourth baseline is an LLM-synthesis pipeline in the style of The Human-AI Hybrid Delphi Model. Experts rate statements and write justifications, and a language model turns the justifications into conditional rules. HAH-Delphi's reported figures are 92.3% single-round consensus and 95% directional agreement. These measure agreement on fixed items, not fidelity of a learned rule. We therefore report them for context and do not treat them as the number to beat.

All methods receive the same total budget of individual expert responses, where one rating of one case counts as one response. They also get the same number of rounds, the same feature schema, the same tree depth limit and the same threshold $\tau$. Baseline hyperparameters are tuned on simulated panels with the same number of trials as ours.

Two ablations isolate the key ingredient. The first is single-expert counterexamples without panel verification, where any expert's counterexample directly updates the tree. The second is panel verification of random cases instead of expert-proposed ones.

Beating the vignette baseline means one of two things. Either the protocol matches its fidelity with at least 40% fewer expert responses, or it reaches higher fidelity at equal budget, especially on recovering the divergent region.

Why not just…?

Why not just survey experts on many random vignettes and fit a tree? This is the real threat. If the rules experts hold are shallow and the feature space is small, random vignettes may cover every branch cheaply, and counterexamples would add nothing. The argument for counterexamples is that consequential exceptions, such as rare comorbidity combinations, occupy a small share of case space. Random sampling finds them at a rate proportional to that share. That is the $\Omega(1/\epsilon)$ behaviour in Exponential Separation between Two Learning Models, whereas an expert who knows the exception can name it directly. The simulated experiments vary exception mass explicitly. If the vignette baseline matches us down to small exception mass, we will report that the protocol's value is limited to rules with rare, high-stakes exceptions, and we will narrow the claim accordingly.

Why not just let an LLM panel produce the consensus? Delphi: Towards Machine Ethics and Norms reports 60.2% zero-shot accuracy for GPT-3 on the Commonsense Norm Bank, against 92.8% for a model fine-tuned on that bank. Aligning AI With Shared Human Values reports drops on adversarially filtered test sets, from 71.0% to 47.9% for ALBERT-xxlarge. HAH-Delphi itself finds that model justifications lack experiential nuance. Those failures sit exactly where counterexamples live. We still use an LLM for a narrow job, drafting minimized cases that experts confirm, and we include an arm in which the LLM also proposes counterexamples. That arm tests whether experts are needed for generating counterexamples or only for judging them.

Why not just use standard active learning on a case pool? Uncertainty sampling only asks about regions the current model already considers uncertain. An exception the tree has never seen gets confident, wrong predictions and is never queried. Counterexample queries target precisely these cases. The ontology-learning result in Actively Learning Concepts and Conjunctive Queries under ELr-Ontologies shows that structural targets need counterexample-driven refinement, not pool uncertainty. The pool-based baseline, run at equal budget, tests whether this matters for clinical rules. Pool active learning may do well when the pool happens to contain exceptions, so we report results separately for pools that do and do not contain them.

Proposed Experiments

The first experiment uses simulated panels, where ground truth is known. We generate target panel concepts as depth-3 to depth-5 trees over 8 to 20 features. Each simulated expert is the panel tree plus a private perturbation, and responses are flipped with rate $\eta$ between 0 and 0.2. We vary the probability mass of exception branches over two orders of magnitude, panel size between 6 and 25, and threshold $\tau$. The metrics are the following: fidelity of the learned three-valued tree to the true panel concept on held-out cases; precision and recall of the divergent region; total expert responses; and responses per expert, which serves as a proxy for burden. The decisive check is the response budget each method needs to reach 95% fidelity, plotted against exception mass. We expect our protocol's budget to grow slowly as exception mass shrinks and the vignette baseline's budget to grow roughly inversely.

The second experiment uses the single-peaked preference testbed of Exact Learning of Preference Structure with $m$ up to 1000. There the noiseless equivalence-query count is exactly $m-1$. We replace the single oracle with a panel of noisy voters, whose axes are perturbed by local swaps. We then measure how many panel-verified counterexamples are needed to recover the majority axis and how accuracy degrades with noise. This is the cleanest place to see whether panel verification preserves the linear query count.

The third experiment is a real clinical study. We run two parallel Delphi arms on one decision rule, for example candidacy for a surgical approach in elderly hip-fracture patients, echoing Chinese expert consensus on the surgical treatment of femoral neck fracture. The arms are a classical static Delphi and our protocol, each with about 8 clinicians, matched for seniority, under an equal response budget. A third, independent clinician panel then rates a held-out set of realistic cases, including cases deliberately built near reported exceptions. Its supermajority labels serve as the reference.

The decisive checks in the clinical study are three. First, whether the learned rule agrees with the independent panel more often than the static Delphi's statements do, on the exception-rich subset. Second, whether the divergent regions our protocol reports are also split in the independent panel. Third, whether attrition and self-reported burden are no worse than in the static arm. If the rule is no more accurate on exceptions, or the divergent regions do not replicate, then the protocol does not deliver what it claims for human panels. Any positive result would then rest on the simulation alone, and we would say so.