Confidence Intervals for Causal Effects When Experts Must Rule Out Hidden Confounders

When an observational study cannot measure every common cause of a treatment and an outcome, the data alone usually cannot pin down the causal effect. Some assumption has to come from outside the data, and in practice it comes from domain experts saying things like "there is no hidden factor driving both drug choice and recovery once we account for disease severity." Today those judgments enter analyses informally. One analyst draws one causal diagram, and that diagram is then treated as true. We propose a different route. A structured panel of experts produces a set of plausible causal diagrams, including hidden-confounder links, together with a record of how much the panel disagreed. We then build a confidence interval for the effect that is valid as long as the true diagram lies somewhere in that set. Whatever part of the experts' claims the data can check is checked and discarded if contradicted. The rest is reported as an explicit, auditable assumption.

The approach combines three things that have so far stayed separate. The first is test inversion: a confidence interval contains every candidate effect value that at least one plausible causal structure fails to reject. The second is the Delphi method, a protocol of anonymous, multi-round expert surveys with controlled feedback, used here to define which structures count as plausible. The third is unobserved confounding, which is the reason experts are needed at all.

Core Claim

Test-inversion confidence regions can be extended from fully observed linear models to linear models with hidden confounders. These are drawn as acyclic directed mixed graphs, where a two-headed arrow between two variables means an unmeasured common cause. Under hidden confounding, most graphs leave the effect unidentified, so an unrestricted region is the whole real line. The only thing that makes the region finite is expert exclusion of specific confounding arrows.

Our central object is

$$C_{\alpha,\tau} = \{\beta : \exists\, G \in \mathcal{E}_\tau \text{ such that the split likelihood-ratio test of } (G,\beta) \text{ does not reject at level } \alpha\},$$

where $\beta$ is a candidate effect value, $G$ a mixed graph, $\alpha$ the test level, and $\mathcal{E}_\tau$ the set of graphs that satisfy every structural statement endorsed by at least a fraction $\tau$ of the panel in its final Delphi round.

Two properties follow. The coverage guarantee is $P(\beta_0 \in C_{\alpha,\tau}) \ge 1-\alpha - P(G_0 \notin \mathcal{E}_\tau)$, where $\beta_0$ and $G_0$ are the true effect and graph. It comes from the union-over-graphs construction, since the true graph's own test is valid. The second property is that an empty region is a statistical falsification of the panel: every graph the experts allowed is rejected by the data. The consensus threshold $\tau$ therefore becomes a tuning knob with a measurable trade-off. Requiring near-unanimity before imposing a constraint gives wider but safer intervals. Accepting majority views gives narrower intervals that are exposed to collective error. Mapping that curve, and showing that keeping minority-endorsed structures buys back coverage cheaply, is the main empirical deliverable.

What Exists, and What This Changes

Test inversion under structure uncertainty is the work of Strieder and Drton. Confidence in Causal Inference under Structure Uncertainty in Linear Causal Models with Equal Variances and Confidence in causal discovery with linear causal models invert constrained and split likelihood-ratio tests over all DAGs. Dual Likelihood for Causal Inference under Structure Uncertainty makes this 10 to 100 times faster through a least-squares reformulation, and Identifying total causal effects in linear models under partial homoscedasticity shows that equal variances between only the cause and the effect suffice for identification. These papers supply our method: we reuse the split likelihood-ratio test, which is valid in finite samples without regularity conditions. They also motivate the gap. All of them assume no hidden confounders, and the dual-likelihood shortcut breaks once error covariances are non-diagonal. Inference of causal effects when control variables are unknown gets valid Wald intervals over near-acyclic graphs by continuous optimization, again assuming no hidden confounding. On the confounding side, Towards Estimating Bounds on the Effect of Policies under Unobserved Confounding bounds policy effects from partial adjustment sets on a known graph and leaves test inversion over graphs to future work. That is exactly the step we take.

Expert input to causal discovery exists, but only in a single-expert, Bayesian form. Expert-Aided Causal Discovery of Ancestral Graphs is the closest work. It samples graphs with hidden-confounder edges using a generative sampler and updates on noisy three-way answers from one human or GPT-4o ("ancestor, hidden confounder, or no relation"). It targets graph recovery, not effect intervals, and offers no frequentist guarantee or test of whether the expert is compatible with the data. Expert-In-The-Loop Causal Discovery: Iterative Model Refinement Using Expert Knowledge proves recovery when a single oracle orients edges with accuracy around 2/3, but rules out hidden confounding. Targeted Causal Elicitation chooses queries for a downstream task, with experts modeled as Bernoulli, Beta or Dirichlet channels over direct edges only. We differ from all three in two ways. We use a multi-expert Delphi panel whose disagreement is carried into the inference rather than averaged away. And we demand coverage that degrades transparently, not silently, when the panel is wrong.

Two further lines of work constrain the design. Falsification before Extrapolation in Causal Effect Estimation is a structural cousin: it takes the union of surviving estimators after falsifying confounded ones, and its guarantee needs at least one unconfounded candidate. Our analogous requirement is that the true graph lie in $\mathcal{E}_\tau$, and $\tau$ lets the user trade that risk against width. The Delphi literature warns that showing panelists the group median can cause artificial convergence (see An international, Delphi consensus study to identify priorities for methodological research in behavioral trials in health research). Because herding produces correlated expert errors, which single-oracle noise models ignore, we test robustness to it explicitly.

Research Program

The first component is the inference engine for linear Gaussian models over bow-free mixed graphs. In such graphs a variable pair never has both a directed edge and a hidden-confounder link, which keeps many effects identifiable. For each candidate graph and hypothesized $\beta$, the constrained maximum likelihood fit is computed on the estimation half of the data, and the split likelihood ratio is evaluated on the test half. Graphs in which $\beta$ is not identified contribute the full line, so they are flagged and never silently dropped. To keep enumeration tractable, we restrict attention to graphs that differ in features relevant to the target effect. Such features are an edge on a path into $X$ or $Y$, or a confounding arc touching their ancestors. We also adapt the sink-first branch-and-bound pruning of the dual-likelihood paper as a heuristic ordering, but we do not rely on its closed form.

The second component is the elicitation protocol. Panelists answer only target-relevant questions. These questions are chosen as the variable pairs whose answer changes identifiability or moves the region most, which borrows the targeting idea from Targeted Causal Elicitation. Each question has the three-way ancestral form used by the ancestral-graph work. There are three anonymous rounds with median-and-rationale feedback, and the endorsement fraction is recorded for every statement and round. The output is not a consensus graph but a family of sets $\mathcal{E}_\tau$ indexed by $\tau$.

The third component separates testable from untestable expert claims. Some excluded confounding arcs imply conditional independences or equality constraints that the data can reject. Others are observationally silent. We report the region together with the list of untestable assumptions it rests on. We also report the smallest $\tau$ at which the region becomes empty, which serves as a data-driven alarm that the panel's view contradicts the data.

Baselines

The naive reference is the plug-in pipeline. We run FCI (a standard discovery algorithm that allows hidden confounders), take its single output graph, and compute ordinary confidence intervals for the effect. It shares the same data and needs no tuning. Prior work shows how badly this can fail: Inference of causal effects when control variables are unknown reports 0% coverage when OLS conditions on a mistaken collider. Beating it means near-nominal coverage at any width.

The bootstrap-over-discovery baseline resamples the data, reruns discovery, and takes percentile intervals. Confidence in causal discovery with linear causal models reports coverage as low as 56% against a nominal 95%, and Confidence in Causal Inference under Structure Uncertainty in Linear Causal Models with Equal Variances reports as low as 70%. We give it the same number of resamples as our method has graph evaluations.

The strongest existing frequentist method is Strieder and Drton's constrained and split likelihood-ratio inversion over DAGs. It reports 95% coverage on synthetic linear models without hidden confounders, in the same two papers. We run it unchanged on data generated with hidden confounders, under identical sample sizes and compute. The point is to show that its coverage, not its width, breaks once the no-hidden-confounder assumption fails. To our knowledge there is no reported result on Lacuna for this method under hidden confounding, so we establish that number.

The strongest expert-using competitor is the ancestral-graph generative sampler with expert feedback from Expert-Aided Causal Discovery of Ancestral Graphs. We feed it the same panel answers, either pooled or one panelist at a time. Its posterior samples feed per-graph effect estimates, from which we form 95% credible intervals. It reports structural Hamming distance gains on Sachs but no scalar value, so there is no reported result on Lacuna for effect-interval coverage. Beating it means equal or better coverage when experts are wrong, at comparable width when they are right.

Two ablations isolate our ingredients. The first replaces the Delphi set with a single expert's graph set, which isolates the role of multi-expert dissent. The second runs our inference with $\tau$ fixed at the conventional 70% consensus cutoff instead of sweeping it, which isolates the value of keeping minority-endorsed structures. Model-agnostic natural bounds are reported as the conservative ceiling on width.

Why not just…?

Why not just elicit a confounding-strength parameter from the panel and run sensitivity analysis? This is the most serious alternative. Sharp Bounds for Generalized Causal Sensitivity Analysis gives fast, sharp bounds given a strength parameter $\Gamma$, and choosing $\Gamma$ is exactly the open calibration problem a Delphi panel could address. However, sensitivity bounds presuppose a known adjustment set among the observed variables. They do not account for uncertainty about which observed variables are colliders or mediators, which is where the plug-in failures above come from. We will settle this empirically. We run sensitivity bounds with a Delphi-elicited $\Gamma$ on the same tasks, with the adjustment set taken from FCI. If those bounds match our coverage and width in the low-dimensional settings, we will say so and restrict our claims to settings with real uncertainty about the observed structure.

Why not just ask one strong LLM or one expert for the graph? Single oracles produce correlated, uncalibrated errors. The ancestral-graph paper had to model GPT-4o's noise explicitly, and Expert-In-The-Loop Causal Discovery needed data-driven pruning to correct its LLM. The single-expert ablation measures how much coverage is lost when that one source is wrong.

Why not just do Bayesian model averaging over expert-weighted graphs? Posterior intervals average over graphs instead of taking a union. When the panel's prior mass sits on the wrong graph, they can be confidently wrong, as meta-analysis was on WHI: Falsification before Extrapolation reports 3% coverage for meta-analysis against 45% for its falsification-based union. The comparison with the generative sampler under injected expert error tests this directly.

Why not just keep only items meeting the usual consensus cutoff? That is our fixed-$\tau$ ablation. If a 70% cutoff already sits at the good corner of the width–coverage curve, the sweep adds little and we will report that.

Proposed Experiments

The decisive synthetic study uses the 6- and 12-variable linear designs of Strieder and Drton. We add one to three latent variables and then marginalize them out, so the true mixed graph is known. Simulated panels of five to fifteen members have controllable individual accuracy and controllable correlation between members, the latter mimicking herding. Correlated errors are what break union-over-dissent less than averaging, so this setting is where we expect the clearest separation. We measure coverage against 95%, median interval width, the fraction of infinite intervals, and the rate at which wrong panels are flagged by empty regions. The success criterion is coverage within two points of nominal whenever the true graph is in $\mathcal{E}_\tau$, with width strictly below natural bounds. A further criterion is that when panel accuracy degrades, the method's coverage loss is smaller than that of the generative-sampler and single-expert baselines.

The real-data checks are Sachs protein signaling (11 variables, with interventional ground truth for selected effects) and the DREAM3 10-node networks. On these we run a three-round Delphi among LLM agents, following An LLM-based Delphi Study to Predict GenAI Evolution, and on Sachs also a small human biologist panel. Comparing the human and LLM panels' $\tau$-curves tests whether panel type matters. On the obesity and lifestyle dataset used by Bellot and Chiappa, we check whether our region agrees in sign with their known-graph bounds once their graph is replaced by a panel-derived set.

The falsification test is the final check. We give panels deliberately corrupted information that excludes a true confounding arc. We then measure how often the data render the region empty, and how often the corruption instead survives as an untestable assumption. The second case is the honest limit of what any data-plus-experts method can promise.