Research on how family and school environments shape children's learning is almost entirely observational. Regression studies show, for example, that self-reported study habits, family education and IQ all predict grades. What they cannot show is which way the arrows point: does a supportive home cause good study habits, or do the two merely co-occur? Recent causal-inference tools can turn uncertainty about a causal diagram into honest bounds on an intervention's effect. Those bounds are only as good as the expert judgments about which arrows are certain, which are impossible, and which are disputed. Nobody has studied where those judgments should come from when experts disagree.
The Delphi method is a structured way to gather expert opinion. Panelists rate items anonymously, see a summary of the group's ratings, and re-rate over several rounds. We propose using a Delphi panel to rate every candidate causal arrow among environmental variables, then treating the panel's remaining disagreement, not its consensus, as the set of graphs over which effect bounds are computed. The empirical question is sharp and has not been asked: do Delphi rounds narrow the effect bounds while still containing the true effect, or does the group feedback make panelists' errors correlated, so that confident but wrong constraints exclude the truth?
Core Claim
Two recent methods give us most of the machinery. Your Assumed DAG is Wrong And Here's How To Deal With It computes upper and lower bounds on an average treatment effect across all directed acyclic graphs (DAGs, causal diagrams without feedback loops) that are consistent with user-supplied "sure," "forbidden," and "contested" edges. It does this through a differentiable relaxation, and it explicitly assumes those labels are handed to it. Expert-In-The-Loop Causal Discovery refines a graph by querying a single expert, human or the Gemini 1.5 Flash language model, for edge orientations. It proves the graph is recovered when the expert is better than chance, but it has no way to reconcile several experts who disagree.
Our claim is that a multi-round anonymous panel is the right source of edge labels, and that the mapping from ratings to labels should be explicit. Each panelist rates on a 1–9 scale the statement "X directly causes Y." An edge becomes sure if the median rating is at least 7 and the interquartile range is at most a threshold $q$. It becomes forbidden if the median is at most 3 under the same spread condition. Every other edge stays contested.
Varying $q$ turns Delphi's notoriously arbitrary "consensus threshold" into a dial with measurable consequences. A loose $q$ fixes more edges and gives narrower bounds at the risk of excluding the truth. A tight $q$ leaves more edges contested and gives wider bounds. We predict that iteration helps only when the feedback between rounds carries information beyond peer opinion. With feedback that only shows peer medians, we expect rounds to shrink the interquartile ranges faster than they improve accuracy, a herding effect that should appear as lost coverage at loose thresholds.
Research Program
The work has three parts.
The first part is a simulation study that tells us what to expect. We generate random structural causal models and simulate panels of 15–30 noisy experts. Each simulated expert has an individual orientation error rate and a "herding" parameter that pulls the next round's rating toward the previous round's median regardless of correctness. We feed the resulting labels to the bounding method of Padh et al. and trace coverage against bound width as rounds and $q$ vary. This maps the regime where iteration pays off and where it backfires, and it fixes the analysis plan before any human panel is run.
The second part is the decisive real-world test. We need a setting where the true effect of an environmental intervention is known and where real experts have genuine domain knowledge. We use the Infant Health and Development Program, a randomized trial of home visits and specialist childcare for low-birth-weight infants, which Padh et al. also use as a benchmark. The semi-synthetic version they report has a simulated outcome surface, and real experts have no special knowledge about a simulated outcome. Our primary test instead uses the randomized estimate as ground truth and builds an observational subsample by non-random removal of treated children. The file does not document the details of this construction, so we will specify and pre-register it ourselves.
Panels of developmental psychologists, early-childhood educators and pediatric nurses then rate edges among the baseline covariates, the treatment and the outcome. They do this in one of three arms. The first arm is classic Delphi, with feedback of medians, interquartile ranges and anonymized rationales. The second arm repeats rounds with no feedback, which controls for the effect of simply reflecting again. The third arm adds data feedback: between rounds, panelists also see the observed data's conditional-independence evidence for each disputed edge, in the spirit of the EXPAND step of Mohammadi et al.
The third part is the application to the dispute in the student-achievement literature. Multivariate Statistical Analysis of Influencing Factors of Student Achievement reports that study habits dominate (standardized β = 0.471) over family education (β = 0.175) and learning environment (β = 0.100). Exploring the Impact of Family Factors on Students English Learning Motivation attributes nearly 58% of motivation variance to family factors. The Analysis of the Positive Patterns in Education argues that the home is the root determinant. These findings are compatible if study habits mediate family effects, and observational regression cannot tell the difference.
On the UCI Student Performance data, which records parental education, family relationship quality, study time, failures, absences and grades, as used in Graph-based Ensemble Machine Learning for Student Performance Prediction, we run a teacher-and-psychologist panel. We then report Delphi-derived bounds on the total effect of parental education on final grades.
Panelists rate fine-grained recorded variables rather than umbrella constructs such as "family education." The reason is that Causal Consistency of Structural Equation Models shows that aggregating variables with opposing effects creates interventions whose outcome is ambiguous. Their example is "total cholesterol," which combines protective and harmful forms.
There is no ground truth on the UCI data. This part is therefore an application, not a validation. Its value is to show whether the published disagreement survives honest uncertainty, meaning whether the bounds exclude zero.
One constraint shapes everything. The bounding method assumes all confounders are observed, and educational data violate this. We handle it in two ways. Panelists may flag suspected hidden common causes. Any variable pair with a flagged hidden cause is excluded from the effect query, and that limitation is reported rather than hidden.
Baselines
The first baseline is the naive reference point. It runs Padh et al.'s bounding with no expert constraints, so every edge is contested, on the same data and with the same optimizer budget. This yields the widest valid bounds. Any expert-informed method must be narrower to be worth anything, and it must keep coverage while doing so.
The strongest existing competitor is the same bounding method fed labels from a single expert. We use two versions. One is the most senior panelist's round-one ratings, which is the realistic way assumed graphs get written today. The other is a language model queried as in Mohammadi et al., where the reported evidence is that an expert with effective orientation accuracy of 2/3 already beats PC, GES and hill-climbing on Structural Intervention Distance, a measure of how many interventional predictions a graph gets wrong. The file gives no absolute Structural Intervention Distance values for that comparison. On the outcome scale, the reference number is Padh et al.'s IHDP bound of [2.4, 7.2] around a true effect of 4.7, with 97–100% coverage on 25-node synthetic graphs.
Our IHDP construction differs from theirs, so we re-run their method on our version rather than quote that interval as directly comparable. A third baseline is the purely data-driven route: a graph estimated by PC or GES, plugged in as if certain, which gives a point estimate with no bounds. For fairness, every arm uses identical data, the same bounding optimizer with the same random restarts and compute budget, and the same $q$ sweep.
The ablations that isolate the key ingredient are the no-feedback and data-feedback arms described above, together with a one-round pooled panel that uses only first-round ratings. Delphi "beats" the single-expert baseline if, at matched coverage of at least 90% across resampled observational subsets, its bounds are meaningfully narrower. Delphi beats the pooled panel only if additional rounds narrow the bounds further without losing coverage.
Why not just…?
Why not just ask one strong expert or a language model, as Mohammadi et al. do? Their guarantee concerns recovering a single graph from an oracle that is right more often than not. It says nothing about calibrated uncertainty. A single expert's confident errors become "sure" edges that can silently exclude the true effect. The single-expert arm measures this directly as coverage loss.
There is also a cultural argument for a panel. CALMA found that US and Indian participants defined identical evaluation terms along different axes. That warns against one perspective on context-dependent constructs such as parental warmth. Still, if the language model matches the panel on IHDP, that is a real, cheap win, and we would report it as such.
Why not just average one round of independent ratings and skip iteration? This is the most serious threat. Independent pooling avoids herding, and herding is exactly the failure we predict for median-only feedback. We do not argue against pooling. It is an arm of the experiment, and the disagreement-as-uncertainty-set mapping applies to it unchanged. If one round matches or beats classic Delphi, the conclusion is that pooled disagreement is enough and that iteration should either be dropped or carry data feedback. That would be a useful negative result about Delphi, not a failure of the project.
Why not just fit a regression or a structural equation model, as the education literature already does? The β coefficients of Tan and Zhu come from a specific, unstated causal ordering. Swapping whether study habits mediate family effects changes what "the effect of family" means without changing the fit. The application arm shows this explicitly. We report the regression coefficient alongside the Delphi-derived bounds on the total effect of parental education, and we show whether the two agree in sign and magnitude.
Why not just run a face-to-face expert meeting? Delphi studies such as the ankle cartilage consensus adopt anonymity specifically to stop authorities from dominating. We do not test open meetings here, because they offer no clean rating-to-label mapping. That omission is a scope choice, not a finding.
Decisive Checks
The central result is a pair of coverage-versus-width curves on the real-trial IHDP construction, one per arm, traced across $q$ and across rounds. The proposal succeeds if some arm with expert input dominates the unconstrained baseline and the single-expert baseline: at least 90% coverage of the randomized effect, with clearly narrower intervals.
The proposal makes a specific, falsifiable prediction about iteration. Median-only feedback should show interquartile ranges shrinking across rounds faster than per-edge agreement with the data-supported orientation improves, and coverage should drop at loose $q$. The data-feedback arm should not show this pattern. The simulation study must reproduce the same qualitative pattern under its herding parameter. If it does not, our explanation of any human-panel result is wrong.
We will also report panel attrition across rounds and analyze completers and dropouts separately. Attrition is Delphi's known weak point, and survivor bias could otherwise masquerade as convergence. Finally, on the UCI data we report whether the bounds on the effect of parental education on grades exclude zero, and whether the bounds on a direct effect that bypasses study time also exclude zero. This tells the education literature whether its family-versus-habits dispute is resolvable from these data at all.
References
- Your Assumed DAG is Wrong And Here’s How To Deal With It
- Expert-In-The-Loop Causal Discovery: Iterative Model Refinement Using Expert Knowledge
- Multivariate Statistical Analysis of Influencing Factors of Student Achievement
- Exploring the Impact of Family Factors on Students English Learning Motivation among English Major Undergraduate Students in China
- The Analysis of the Positive Patterns in Education
- Graph-based Ensemble Machine Learning for Student Performance Prediction
- Causal Consistency of Structural Equation Models
- CALMA: Context‑Aligned Axes for Language Model Alignment
- Conservative Management and Biological Treatment Strategies: Proceedings of the International Consensus Meeting on Cartilage Repair of the Ankle