When scientists need a causal diagram and the data cannot settle it, they often convene expert panels. The Delphi method, an anonymous, multi-round survey where panelists see the group's summary statistics and each other's rationales between rounds, is the standard tool for this in medicine and public health. Delphi treats each question as a separate item, though. A panel can reach 80% agreement on every individual causal claim while the claims together form a loop, or imply contradictory statements about which variables are independent. The panel also spends rounds debating arrows that have no bearing on the decision it was convened to inform. We propose a Delphi protocol whose between-round feedback is computed by a graph engine. Before each round, the engine takes the panel's current answers, finds the causal diagrams still compatible with them, and reports two things back to the panel. It flags which agreed-upon claims are jointly impossible. It also identifies which remaining disagreements would actually change the valid way to estimate the target causal effect. Later rounds cover only those disagreements.
The engine is built from two classical tools. The first is d-separation, the rule that reads off from a directed causal diagram which variables become statistically independent once others are held fixed. The second is graph moralization, the step that "marries" any two causes sharing a common effect and then drops the arrow directions, which turns d-separation into ordinary graph cutting. Moralization makes each compatibility and relevance check take linear time, so the engine can re-evaluate thousands of candidate diagrams between rounds. The claim to test is that, at an equal budget of expert effort, query-targeted feedback produces consensus diagrams that support correct causal adjustment more often than standard edge-voting Delphi, including when that Delphi's output is repaired afterward into a valid diagram.
Core Claim
Expert disagreement about a causal diagram is mostly irrelevant to any single causal question. The relevant part can be computed exactly and cheaply. Suppose a panel is asked for the effect of $X$ on $Y$. By the constructive back-door criterion of Constructing Separators and Adjustment Sets in Ancestral Graphs, a set $Z$ is a valid adjustment set exactly when $Z$ separates $X$ from $Y$ in the moralized "proper back-door graph." Checking this takes $O(n+m)$ time for $n$ variables and $m$ edges. So for any disputed item $q$, we can sample diagrams consistent with the panel's current answers and ask whether the family of valid adjustment sets differs between the diagrams that answer $q$ "yes" and those that answer "no." We call this the item's decision relevance. Our proposal is that Delphi feedback should report decision relevance and joint consistency alongside the usual medians and rationales. It should also retire items whose relevance is zero. Two consequences follow. Rounds should converge faster on the parts that matter. Panelists should also face less pressure to conform on the parts that do not matter, which addresses the known Delphi failure where people drift toward the median simply to end the process.
A second design choice follows from the engine. Panelists can state conditional-independence claims and ancestral exclusions directly, not only pairwise arrows. Examples are "once we know the biomarker, the symptom tells us nothing more about history" and "A cannot affect B by any route." The engine converts these into constraints on the diagram through the ancestral moral graph. It does not need them to be phrased as edges.
Position Relative to Prior Work
The closest work elicits causal structure from experts but never lets a panel deliberate. Targeted Causal Elicitation asks a single expert pairwise edge questions. It models the answers with Bernoulli, Beta, or Dirichlet likelihoods inside differentiable Bayesian structure learning, and picks the next question by expected information gain about a downstream causal query. We share its query-targeting idea and reuse its expert likelihoods. We differ in having many experts who revise their answers after seeing structured feedback, in accepting independence and ancestral statements, and in measuring relevance through exact moral-graph adjustment checks instead of a continuous posterior over adjacency matrices. Your Assumed DAG is Wrong And Here's How To Deal With It takes expert disagreement as fixed "sure" and "forbidden" edges and bounds the treatment effect over all compatible diagrams with gradient-based search. That is the right output format for a panel that cannot agree, and we use it as our endpoint. It has no feedback loop to shrink the disagreement, and it assumes no hidden confounders. Large Language Models as Tools to Improve Bayesian Causal Discovery replaces experts with repeated LLM prompting over edge pairs. It motivates using LLM panelists as a testbed, and it constrains how we use them, since it warns that benchmarks like Sachs are likely memorized.
The graph machinery comes entirely from existing work. The reduction of d-separation to separation in the ancestral moral graph, and the minimum-separator construction of An Algorithm for Finding Minimum d-Separating Sets in Belief Networks, supply the consistency checks. Decomposition of search for v-structures in DAGs shows that clique separators of the moral graph localize independence questions. On ALARM, the 37-vertex moral graph splits into 23 subgraphs, the largest with 7 vertices. We use this to split a large survey into sub-surveys that smaller subpanels can answer independently. Approximate implication with d-separation proves that independence statements in the "local" form preserve approximation error with factor 1, while marginal statements can amplify it by $|X|\cdot|Y|$. It concerns information measures of distributions, not beliefs, so we use it only as a design heuristic. We prefer items phrased as "given its direct causes, X is independent of Y," and we test whether panelists answer such items more reliably. Finally, Structural Intervention Distance (SID) for Evaluating Causal Graphs shows that counting edge differences is blind to whether a graph supports correct interventions. That result motivates scoring consensus by intervention-level error rather than edge counts.
Research Program
The engine represents the panel state after round $r$ as a set of weighted constraints. Each panelist's answer to each item contributes a likelihood term in the style of Targeted Causal Elicitation, generalized from edges to independence and ancestral statements. For the benchmark sizes we use (7 to 37 variables), we draw diagrams from the resulting distribution with order-based Markov chain Monte Carlo. Each sampled diagram is checked against every item by moral-graph separation. For the target pair $(X,Y)$, we compute each item's decision relevance as
$$R(q) = \Pr_{G}\big[\mathcal{A}(G)\neq\mathcal{A}(G')\big], \quad G \sim \pi_r(\cdot \mid q=\text{yes}),\; G' \sim \pi_r(\cdot \mid q=\text{no}).$$
Here $\pi_r$ is the round-$r$ distribution over diagrams, and $\mathcal{A}(G)$ is the minimal valid adjustment set that the van der Zander construction returns in $G$ (or "not adjustable"). Items with $R(q)$ below a small threshold are retired and shown to panelists as settled-for-this-question. Items that are jointly inconsistent with high-agreement items are returned with a short explanation of the conflict. An example explanation is "you agreed A and B both cause C and are independent; you also agreed they are independent given C; these cannot all hold."
Panelists still see the usual medians and anonymous rationales. The graph-derived feedback is added to them and does not replace them. When the panel stops converging, the protocol ends by passing the remaining constraints to the bounding approach of Padh et al., so the panel's output is an effect interval rather than a forced single diagram.
Baselines
The naive reference point is standard edge-voting Delphi. Each ordered variable pair is an item, three rounds are run with median and interquartile feedback, and an edge is kept at 75% agreement. The output is scored as-is, cycles and all, which shows how often plain Delphi produces an unusable diagram.
A stronger version of the same baseline adds post-hoc repair. The edge-voting consensus is projected to the nearest acyclic diagram consistent with the vote weights, and adjustment sets are computed from it. This is the fairest human-protocol competitor, because it gets the same final consistency guarantee as our method without changing what panelists see.
The strongest existing elicitation method is Targeted Causal Elicitation, run once per panelist with its active pairwise queries. The answers are pooled by multiplying likelihoods, and the output is the posterior's adjustment set. The file contains no reported numerical result on Lacuna for it. It is the strongest competitor because it already targets the causal query, and our advantage must therefore come from deliberation and richer items, not from targeting alone.
For the bounding endpoint, the reference is Padh et al.'s bounds computed from the round-1 constraints without further rounds. On IHDP they report bounds of [2.4, 7.2] around a true effect of 4.7, with 97–100% coverage in simulation. We beat it if our post-deliberation constraints give strictly narrower intervals at equal or higher coverage.
A data-only reference is also included. For Sachs, OSLow reports a structural Hamming distance of 10.4 (OCDaf: Ordered Causal Discovery with Autoregressive Flows). FASK reports adjacency precision 0.84 and recall 0.80 (FASK with Interventional Knowledge Recovers Edges from the Sachs Model).
All protocols get the same panel and the same total budget of item-answers per panelist, and the same diagram sampler where one is used.
Three ablations isolate the key ingredient. The first gives consistency feedback without relevance-based retirement. The second gives relevance-based retirement without consistency feedback. The third keeps our full feedback but restricts items to pairwise edges.
Why not just…?
Why not just run edge-voting Delphi and repair the output afterward? This is the most serious threat. Repair fixes consistency but does nothing about what the panel spends its attention on, and repair silently chooses among conflicting agreed claims without asking anyone. Whether that matters empirically is exactly what the repaired-Delphi baseline tests. If repair matches our protocol on adjustment validity at equal budget, then the value of in-loop feedback is refuted. In that case the contribution reduces to richer item types, which the edges-only ablation measures separately.
Why not just pool experts with Targeted Causal Elicitation and skip deliberation? Pooling treats disagreements as independent noise. Delphi's premise is that rationales change minds, for example when one panelist knows about a mediator the others overlooked. If the panelists' errors are uncorrelated, pooling should win on cost. If they share misconceptions, pooling reinforces them. We test both regimes by injecting shared versus private errors into simulated panels.
Why not just prompt an LLM for the diagram? Bazaluk et al. report that GPT priors did not beat an uninformed prior on the less-familiar Summer dataset, while DeepSeek priors did. They also warn of memorization on Sachs. So LLM output is unreliable exactly where panels are needed. We include single-shot LLM priors as a baseline, and we use LLMs as panelists only for studying protocol dynamics, never as ground-truth-free evidence.
Proposed Experiments
The first testbed uses simulated panels on ALARM and Sachs. Each synthetic expert holds the true diagram corrupted by private errors (random edge flips) and shared errors (a common false mediator or a missing collider). Each updates toward cited rationales according to a fixed rule, with a tunable conformity weight that pulls answers toward the median regardless of content. The primary metric is the fraction of treatment–outcome pairs, drawn at random, for which the consensus returns a valid adjustment set in the true diagram. We also report the structural intervention distance, the rate of inconsistent consensus, and the answers spent. The decisive check is at matched budget. The full protocol must beat repaired edge-voting Delphi on adjustment validity. And under high conformity, the full protocol should drift less on retired items, which would show that it removes pressure where no evidence exists.
The second testbed uses LLM panels, with five models or prompt personas per panel, on the Summer dataset and on ALARM with variable descriptions. These panels revise answers with rationales, so we can test whether graph-derived feedback changes their reasoning rather than just their votes. The same metrics apply, and memorization is controlled by comparing against a relabeled-variable condition.
The third testbed is IHDP, where the endpoint is the effect interval. We measure interval width and coverage against Padh et al.'s [2.4, 7.2] from round-1 constraints.
A small human pilot with clinicians on an ALARM sub-survey, formed by splitting ALARM's moral graph along its clique separators, tests two practical points: whether people can answer independence items reliably, and whether local-form items beat marginal ones.
The idea fails if decision relevance proves unstable across samples of diagrams, or if repaired edge-voting Delphi matches our protocol. Either outcome would be informative about how much structure expert feedback needs.
References
- Constructing Separators and Adjustment Sets in Ancestral Graphs
- Targeted Causal Elicitation
- Your Assumed DAG is Wrong And Here’s How To Deal With It
- Large Language Models as Tools to Improve Bayesian Causal Discovery
- An Algorithm for Finding Minimum d-Separating Sets in Belief Networks
- Decomposition of search for<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" altimg="si1.gif" overflow="scroll"><mml:mi>v</mml:mi></mml:math>-structures in DAGs
- Approximate implication with d-separation
- Structural Intervention Distance (SID) for Evaluating Causal Graphs
- OCDaf: Ordered Causal Discovery with Autoregressive Flows
- FASK with Interventional Knowledge Recovers Edges from the Sachs Model