Many causal questions in medicine and biology ask what would happen to an outcome if we intervened on a treatment. Such questions can only be answered from observational data if the causal diagram, the map of what causes what including hidden common causes, has the right shape. A mature theory says exactly which shapes work. Pearl's do-calculus, a set of three rules for rewriting interventional probabilities, is complete: an effect is computable from data if and only if the diagram contains no "hedge," a specific tangle of hidden confounding around treatment and outcome. In practice nobody knows the diagram. Data alone recovers it only up to a family of equally plausible graphs, and the remaining gaps are usually filled by asking experts, who disagree. The Delphi method addresses that disagreement. It is a protocol of anonymous, repeated survey rounds in which experts see summarized peer answers and reasons, then revise. Delphi currently surveys clinical recommendations, not causal structure, and its consensus statements are never checked against graph logic.
This note proposes an identifiability-targeted Delphi protocol. Before each round, a solver works out which few structural facts actually decide whether the target effect is computable, and which formula would compute it. The panel is asked only about those facts. Between rounds, the solver shows experts the consequences of the group's current answers: "if no hidden common cause links A and B, the effect is identified by this formula," or "these three majority answers jointly imply a cycle." The output is not a full consensus graph. It is a conditional estimand: one formula, together with the minimal set of assumptions it depends on and the level of panel agreement behind each assumption. The bet is that most of a causal graph is irrelevant to a given query. Spending expert attention on the decisive parts should reach correct identifiability verdicts with far fewer questions, and with far fewer dangerous false claims that an effect is identifiable.
Core Claim
Fix a target effect $P(y \mid do(x))$ and a set $\mathcal{G}$ of graphs that remain consistent with the data and with the assumptions agreed so far. A candidate structural feature $f$ is any yes/no question an expert could answer. Examples are "is there a hidden common cause of $A$ and $B$," "is $A$ an ancestor of $B$," or "is $B$ independent of $A$ given $C$ in stratum $c$." A feature is decisive if graphs in $\mathcal{G}$ that agree on $f$ cluster by identifiability verdict, or by identifying formula, differently from graphs that disagree on it. We score features by how much they are expected to shrink disagreement over the answer to the query, not disagreement over edges:
$$ \text{score}(f) = H\big(E(G)\big) - \mathbb{E}_{a \sim p(f)}\, H\big(E(G) \mid f = a\big), $$
Here $G$ is drawn from the current weighting over $\mathcal{G}$. $E(G)$ is the output of the complete identification algorithm on $G$: either a symbolic formula or "not identifiable." $H$ is entropy over those distinct outputs. $p(f)$ is the panel's current answer distribution. The claim is that querying in order of this score, with solver-generated consistency feedback between rounds, gives correct identification decisions at a fraction of the question budget that edge-level or graph-level elicitation needs. The largest gains should come in the regime where hidden confounding creates candidate hedges.
Relation to Prior Work
The identification machinery comes from complete results. The first is the ID algorithm and the hedge criterion of Shpitser & Pearl, with polynomial-time testing going back to Galles & Pearl. The second is its extension to arbitrary collections of experiments, via the gID algorithm and its "thicket" obstructions, in General Identifiability with Arbitrary Surrogate Experiments. We reuse these algorithms unchanged as the function $E(G)$.
The closest technical ancestor is Do-calculus when the True Graph Is Unknown. That work encodes data-derived independence constraints in a logic solver, searches the resulting family of graphs for hedges, and returns a formula only if it is the same across the whole family. It reports that the fraction of identified effects falls as graphs get denser. That is our motivation: when data cannot settle the family, the method simply gives up, and it has no way to bring in human judgment. We keep its solver but add an elicitation loop that narrows the family exactly where it matters.
Constrained Identifiability of Causal Effects shows that local facts can make hedged effects identifiable. Examples are determinism, or independence within a specific context, used through compiled arithmetic circuits. It assumes those facts are simply given. Our query space includes such context-specific questions aimed at the nodes inside a detected hedge, which gives that theory a source of constraints.
Four lines of work bring experts into causal discovery. All four optimize graph accuracy rather than query answerability, and none runs Delphi-style rounds. Targeted Causal Elicitation picks pairwise edge questions for a single expert by expected information gain over graph structure. Its acquisition rule is the methodological template we adapt. The differences are that our information target is the identifying formula, that we allow hidden-confounder and context-specific questions, and that we use a panel. Expert-Aided Causal Discovery of Ancestral Graphs asks humans or language models ancestral and confounding questions under latent variables and evaluates by structural Hamming distance, a count of edge mistakes. Expert-In-The-Loop Causal Discovery proves robustness to an oracle that is correct about two-thirds of the time, but it assumes no hidden confounders. That assumption removes hedges entirely. Causal Learning Should Embrace the Wisdom of the Crowd aggregates crowd answers about variable ordering and reports large variance across queries in a 20-person pilot on the Asia network. That result supports aggregation, and also warns that individual experts are unreliable.
On the Delphi side, consensus studies such as Dombrowski et al. and Skrobot et al. supply the procedure we borrow: anonymity, repeated rounds, controlled feedback, and preset agreement thresholds of 51/75/100% and 67% respectively. Dombrowski et al. also note that such consensus is low-grade evidence. That is a constraint on our design: we report agreement levels next to each assumption instead of promoting consensus to truth.
Research Program
The system has three parts. The first is a candidate generator. It runs a latent-variable discovery algorithm on the observational data, takes the resulting partial ancestral graph, and encodes the remaining freedom as solver constraints, following Hyttinen et al. The second is a query selector. It samples graphs from the family, weighted by current panel beliefs. It runs ID or gID on each sample, groups the samples by output formula, and scores candidate questions by the equation above. A short list of top-scoring questions forms the next round.
The third part is a feedback compiler that runs between rounds. It aggregates the panel's answers using two rules: a weighted vote that discounts experts whose earlier answers conflicted with data-derived constraints, and the Delphi threshold. It then checks the aggregate for logical contradictions such as cycles or incompatible independencies. Finally, it translates the state of the family into plain-language feedback. That feedback states which answers currently decide identifiability, which formula each branch implies, and which anonymous reasons were given for the minority answers. The protocol stops when the identification output is the same across all graphs consistent with assumptions that cleared the threshold, or when the question budget runs out. In the second case it reports the effect as not identified together with the decisive unresolved disagreement.
The analysis has one theoretical component. When can the decisive feature set be small? We conjecture it is bounded by the features touching the confounded components, the groups of variables linked by hidden common causes, that sit around the minimal hedges or thickets across the family. If that holds, cost should grow with confounding near the query rather than with graph size. We will test this empirically and attempt a proof for the observational case.
Baselines
The naive reference point is to ask the panel nothing. We run the Hyttinen et al. solver on the data-derived family and report the fraction of queries it identifies and whether those formulas are correct. Every elicitation method must beat this on identified fraction without raising the false-identification rate.
The first strong competitor is full-graph consensus followed by identification. Each expert answers every pairwise question: edge, direction, hidden confounder. Answers are aggregated by majority vote with acyclicity repaired, and ID is run on the result. This is what a practitioner would do with a standard Delphi on a graph.
The second is Targeted Causal Elicitation's information-gain edge querying. We extend it with confounder questions and with the panel averaged into one noisy oracle. It is the strongest competitor because it already targets a downstream task and is query-efficient. The extended confounder questions we add to it are themselves our modification, so we will report the unmodified edge-only version as well.
The third is the ancestral-question protocol of Expert-Aided Causal Discovery, again followed by ID.
All methods get the same data, the same simulated or real panel, the same question budget, and the same aggregation threshold. Hyperparameters such as sample counts and the noisy-oracle reliability are tuned on a held-out set of synthetic graphs with an equal tuning budget per method.
Three ablations isolate the key ingredients. The first replaces our score with information gain over edges, keeping everything else fixed. The second removes iteration: one round with all budgeted questions. The third removes the consistency feedback but keeps the rounds.
The file reports no prior result on identifiability-verdict accuracy or formula correctness under expert elicitation. There is no reported result found on Lacuna for any baseline on our primary metrics. The nearest numbers are structure metrics. On Sachs, adjacency precision is 0.84 and recall 0.80 for FASK with Interventional Knowledge, and structural Hamming distance is 10.4 for OSLow in OCDaf. In synthetic settings, Expert-In-The-Loop reports that an oracle with about 2/3 effective orientation accuracy beats automated methods on intervention distance. We will report these structure metrics for comparability. "Beating" a baseline means matching or exceeding its correct-verdict rate at no more than half the question budget, with a false-identification rate no higher.
Why not just…?
Why not just run data-driven discovery and skip experts? That is the "ask nothing" baseline. Hyttinen et al. already show that identification under data-only uncertainty falls off as density grows. Our experiments quantify how many of those failures a handful of decisive questions can recover.
Why not just ask one strong language model, or one expert, to draw the whole graph? The Wisdom-of-the-Crowd pilot found strong variance across individuals. Expert-Aided Causal Discovery describes language-model answers as helpful but noisy. A single answer risks a confidently wrong formula. We include a single-model, whole-graph baseline. If it matches us on correct verdicts at similar false-identification rates, the panel machinery is not earning its cost, and we would say so.
Why not just a single round of the identifiability-targeted questions, with majority vote? This is the real threat. The selection score may deliver most of the gain on its own, and iteration may add little. The no-iteration ablation settles it directly. If one round matches the full protocol, the contribution becomes the query selector, and the Delphi component reduces to an optional robustness layer that we report as such. Our hypothesis for why iteration should help is narrower: the consistency feedback should fix answers that are each plausible but jointly contradictory, which a single vote cannot do. We measure how many contradictions each round resolves.
Why not just compute partial-identification bounds instead of eliciting? Bounds remain available when the protocol ends unidentified. But hedged effects often have wide bounds, and the identification theory says exactly which extra assumption would collapse them. We report bound width under the "ask nothing" condition next to point estimates after elicitation, so readers can see what the assumptions bought.
Proposed Experiments
The first testbed is synthetic semi-Markovian graphs of 5 to 25 nodes, generated as in Hyttinen et al., with planted hedges and front-door structures. Panels are simulated: 10 to 30 experts with heterogeneous accuracy spanning the 0.5 to 0.8 range suggested by Expert-In-The-Loop, correlated errors within subgroups to mimic shared bias, and a revision model in which low-confidence experts move toward reasoned minority answers. Because the true graph is known, we measure verdict accuracy, formula correctness, false-identification rate, number of questions, and the error of estimates against true interventional distributions.
The second testbed is Sachs, whose interventional conditions give a real check. We estimate effects from the observational subset using the elicited formula and compare against the matching interventional subsets. The panel is a pool of differently prompted language-model agents, together with a small group of human biologists if recruitment allows.
The third testbed is Asia with selected variables hidden to create confounding. This reuses the 8-node crowd setting of Lin et al. and makes a small human pilot of the full multi-round protocol feasible.
The decisive check is the false-identification rate under correlated panel bias on synthetic graphs. If targeting expert attention makes the protocol confidently wrong more often than full-graph consensus, the approach fails regardless of query savings. The second decisive check is the scaling test: whether question count tracks confounding near the query rather than total node count as graphs grow from 5 to 25 nodes.
References
- Identification of Joint Interventional Distributions in Recursive Semi-Markovian Causal Models
- Testing Identifiability of Causal Effects
- General Identifiability with Arbitrary Surrogate Experiments
- Do-calculus when the True Graph Is Unknown
- Constrained Identifiability of Causal Effects
- Targeted Causal Elicitation
- Expert-Aided Causal Discovery of Ancestral Graphs
- Expert-In-The-Loop Causal Discovery: Iterative Model Refinement Using Expert Knowledge
- Causal Learning Should Embrace the Wisdom of the Crowd
- Conservative Management and Biological Treatment Strategies: Proceedings of the International Consensus Meeting on Cartilage Repair of the Ankle
- The Vascular Impairment of Cognition Classification Consensus Study
- FASK with Interventional Knowledge Recovers Edges from the Sachs Model
- OCDaf: Ordered Causal Discovery with Autoregressive Flows