A retrospective cohort study estimates a treatment's effect from historical hospital records. Its answer depends heavily on choices the analyst makes before any estimation happens. These include when the clock starts for each patient, which variables count as confounders, what causal diagram is drawn, and which estimator is fit. Reasonable analysts make different choices and get different answers. Re-analyses with g-methods of one lung-cancer trial produced cumulative hazard ratios anywhere from 0.38 to 0.73 depending only on how the weighting model was specified (Addressing treatment switching in the ALTA-1L trial with g-methods). The field's current response is to show this spread, a "vibration analysis," and leave the reader to interpret it (Step-by-step causal analysis of EHRs to ground decision-making).
This note proposes a different step. Many analysis choices make predictions that the data can check. A causal diagram, or DAG (a directed acyclic graph drawing which variables cause which), implies that certain variables should be statistically independent once others are held fixed. A choice of time zero implies that treatment cannot affect outcomes that happened before it. A confounder set implies that a known "negative-control" outcome, one the treatment cannot plausibly cause, should show no effect. We propose to enumerate the space of analyses jointly over study design, graph, and estimator. Each analysis is run against the checks its own assumptions imply, and only the analyses that survive are reported. The claim to test is that this narrows the spread of estimates toward the truth. We also test the sharper claim that the most popular data-driven alternative, choosing the analysis that best predicts the outcome, systematically moves toward the wrong answer.
Core Claim
In retrospective electronic health record (EHR) cohorts, the analysis choices that cause the largest bias tend to leave detectable fingerprints. Misplaced time zeros produce immortal-time bias, the spurious benefit that appears when treated patients must survive long enough to be treated. Adjusting for a mediator or collider distorts the effect: a mediator lies on the path from treatment to outcome, and a collider is a common effect of treatment and outcome. Both errors are hard to see in outcome prediction accuracy but easy to see in design-implied falsification tests.
We therefore expect three results. First, filtering specifications by falsification tests should shrink the envelope of estimates and improve its coverage of the true effect. Second, selecting specifications by outcome predictive loss should often do worse than no selection at all, because post-treatment variables usually improve prediction of the outcome. Third, disagreement among graph-discovery algorithms should matter less than design choices. The Step-by-step EHR paper observed this informally, and we aim to quantify it.
The new element is not the multiverse itself. It is a scored, falsification-based pruning of a multiverse that spans design, graph, and estimator together, evaluated against known ground truth.
Relation to Prior Work
Two strands of work motivate the proposal by exposing the spread of estimates without resolving it.
The Step-by-step EHR framework uses target trial emulation, which designs the observational analysis as if it were the randomized trial one would ideally run. It combines this with an expert DAG, AIPW (augmented inverse-probability weighting) estimators, and vibration analysis, applied to albumin in sepsis on MIMIC-IV. It finds that design flaws inflate apparent benefit more than DAG perturbations do, but it relies on a single expert graph and offers no rule for discarding specifications. The ALTA-1L re-analysis documents the same "forking paths" problem for IPCW (inverse probability of censoring weighting) and the parametric g-formula, again without an arbitration mechanism.
A second strand supplies graph candidates but stops short of effects. From Prediction to Structure runs three discovery algorithms on a MIMIC-IV readmission cohort of 56,300 patients and 50 features: NOTEARS (continuous optimization), FCI (constraint-based and tolerant of latent confounders), and DECI (Bayesian). It reports a consistent local neighborhood but explicitly does no effect estimation, and it collapses time into static averages. We reuse these three algorithms as generators of candidate graphs. Unlike that paper, we keep the time stamps and carry each graph through to an effect estimate.
Several papers supply methods we reuse or compare against. Causal Foundation Models Perform Better without Post-treatment Variables shows that Do-PFN and CausalPFN condition on every feature, as S-learners do. These are transformer models pretrained on synthetic causal models to output effects in one pass. The paper introduces TC-LD, a conditional-independence filter that prunes post-treatment variables. HTE-FS prunes treatment descendants for heterogeneous-effect estimation. Both methods prune variables within a single cross-sectional table. We instead prune whole analyses, including the design choices those methods cannot see.
Proximal Causal Inference With Text Data is the closest precedent for our key ingredient. It halts inference when an odds-ratio test between two text-derived proxies fails. We generalize that "test before you estimate" idea from one proxy assumption to the full specification space.
One paper constrains the design rather than supplying it. Hyperparameter Tuning for Causal Inference with Double Machine Learning finds that outcome predictive loss picks the correct structural model in over 75% of heterogeneous-effect simulations. It also finds that untuned defaults bias estimates and that nuisance models should be tuned on the full sample or on folds. We adopt that tuning protocol for every specification. We take its selection rule as the main competitor, and we argue it breaks exactly when candidate covariates include post-treatment variables, a setting the paper did not study.
Research Program
The object we build is a specification space for a given clinical question. Each specification $s$ fixes a design, an adjustment set, and an estimator.
The design fixes time zero, the baseline look-back window, the grace period for treatment initiation, and the handling of switching. The adjustment set is derived from a graph. That graph is either the expert DAG, the graph output by NOTEARS, FCI, or DECI on time-stamped features, or a kitchen-sink set containing every covariate. The estimator is one of AIPW with tuned forests, double machine learning (DML), IPCW or the g-formula when treatment varies over time, or CausalPFN and Do-PFN via the Do-PFN code.
Each specification comes with the checks its own assumptions imply. These checks fall into three families.
The first family is design checks. We estimate a placebo effect of treatment on outcomes in windows before treatment could act; a nonzero effect signals immortal time or baseline misalignment. We also check that no baseline covariate is timestamped after time zero.
The second family is graph checks. These are local conditional-independence tests implied by the chosen graph among the adjusted variables, with the proximal-text odds-ratio check where text proxies are used.
The third family is confounding checks. These are effects on negative-control outcomes, which clinicians agree the treatment cannot cause.
Every test yields a p-value. We retain the specifications that survive all their tests after false-discovery correction and report the range and median of their estimates:
$$S^{*} = \{\, s : \min_{k} \tilde p_{s,k} > \alpha \,\},$$
Here $\tilde p_{s,k}$ is the corrected p-value of test $k$ for specification $s$, and $\alpha$ is the rejection level. A softer variant weights specifications by their test evidence rather than cutting them off.
The analysis has three parts.
The first part is a decomposition of estimate variance into design, graph, and estimator components before and after filtering. This quantifies the Step-by-step paper's qualitative observation.
The second part is a power study. It asks which classes of error the tests can and cannot detect. Unmeasured confounding that touches no negative control is the obvious blind spot, and we will report it rather than hide it.
The third part is the main comparison. It tests whether passing the falsification tests correlates with low actual error when the truth is known. If it does not, the method fails, and that is a publishable negative result.
Baselines
The naive reference is the kitchen-sink analysis: a single target-trial design, all available pre- and post-time-zero covariates, and tuned AIPW. This is what a practitioner running high-capacity models under assumed ignorability would do. It is the floor, and any method must beat it.
The second baseline is the expert-graph analysis of Step-by-step causal analysis of EHRs. It uses one specialist DAG and one design, together with that paper's unfiltered vibration envelope, which we summarize by its median and its range. This is the current best clinical practice for exactly our question. We beat it if our filtered envelope is narrower and covers the truth at least as often.
The third baseline is predictive-loss selection from the DML tuning study. Across the same specification space, it picks the specification whose nuisance outcome model has the lowest cross-validated loss. That study reports correct structural-model selection in more than 75% of heterogeneous scenarios, which makes this the strongest data-driven competitor.
The fourth baseline is consensus-graph adjustment. It keeps only the edges on which NOTEARS, FCI, and DECI agree, following From Prediction to Structure, and then runs AIPW on the resulting adjustment set.
On cross-sectional semi-synthetic benchmarks we add TC-LD followed by CausalPFN. The reported strong figure there is 85.5% to 92.6% recovery of the error reduction achievable by oracle removal of post-treatment variables (Causal Foundation Models Perform Better without Post-treatment Variables).
Two ablations isolate the key ingredient. The first uses uniform weighting over the full space, which removes filtering entirely. The second uses each test family alone: design-only, graph-only, and negative-control-only.
All methods share the same cohorts, the same candidate covariates, and the same full-sample-or-on-folds nuisance tuning with an equal budget per specification. Beating a baseline means lower absolute bias of the reported point estimate and at least equal coverage by the reported interval, averaged over simulation replicates.
Why not just…?
Why not just report the vibration envelope, as the Step-by-step paper does? An envelope from 0.38 to 0.73, like ALTA-1L's IPCW range, gives a clinician no decision. Reporting the envelope does not tell the reader whether the wide end reflects real uncertainty or known design errors. The uniform-weighting ablation is exactly this alternative. If filtering does not move coverage or width relative to it, the proposal adds nothing, and we would say so.
Why not just select the specification with the best outcome prediction? This is the real threat, since the DML tuning study found it works in over 75% of its scenarios. But those scenarios contained no post-treatment candidates. In EHR data, a mediator such as a lab value measured after treatment predicts the outcome well, so the lowest-loss specification will tend to adjust for it. That is the mechanism behind the mediation-masking and pneumonia–asthma paradoxes described in The Risks of Risk Assessment. Our semi-synthetic cohorts inject mediators and colliders with known effects, so the comparison settles directly whether predictive-loss selection chooses them.
Why not just drop every variable timestamped after time zero, and skip graphs? Time stamps do remove many post-treatment variables for free, and we include this as a design rule in every specification. They cannot, however, detect a wrong time zero itself, which is where the largest bias appears in the albumin study. They also cannot detect pre-treatment colliders. The design-only ablation measures how much of the gain time ordering alone captures. If it captures nearly all of it, the graph checks are unnecessary, and the paper becomes a paper about design checks.
Why not just use a consensus of discovery algorithms? Agreement among NOTEARS, FCI, and DECI shows robustness to algorithm choice, not correctness. All three can agree on a graph that sits downstream of a biased design. The consensus baseline tests this directly.
Proposed Experiments
The primary testbed is semi-synthetic longitudinal cohorts built on real MIMIC-IV covariate trajectories. We use real patients and timestamps, with simulated treatment assignment, time-varying confounding, injected mediators and colliders, delayed-initiation windows that create immortal time, and known effects. This follows the logic of IHDP and RealCause, where covariates are real and outcomes are generated so the truth is known. The metrics are the bias of the reported estimate, the coverage and width of the surviving envelope, and the rank correlation between each specification's test evidence and its true error.
The decisive check is whether predictive-loss selection loses to falsification filtering specifically in the injected-mediator conditions while roughly tying elsewhere. That pattern would confirm the mechanism. A uniform tie would refute it.
The secondary testbeds are IHDP and ACIC 2016 with injected mediators, where we compare against TC-LD with CausalPFN on PEHE, the error of individual-level effect estimates. We do not expect to beat TC-LD at variable pruning there. We only need to show the filter is not worse when design is held fixed. We also re-analyze the ALTA-1L synthetic individual-level data to test whether design and negative-control checks shrink the range of 0.38 to 0.73 cumulative hazard ratios.
The real-data check is the albumin-in-sepsis question on MIMIC-IV. Randomized-trial meta-analyses there indicate no average benefit, and the Step-by-step paper reports agreement with them qualitatively. We found no reported numeric average effect on Lacuna for this cohort. Success means the surviving envelope contains the null while the rejected specifications are concentrated among those with long pre-intervention windows. Failure means the surviving set still shows inflated benefit, which would indicate the tests lack power for the dominant bias.
References
- Addressing treatment switching in the ALTA-1L trial with g-methods: exploring the impact of model specification
- Step-by-step causal analysis of EHRs to ground decision-making
- From Prediction to Structure: Causal Discovery for 30-Day Readmission Prediction
- Causal Foundation Models Perform Better without Post-treatment Variables
- Improving Data-driven Heterogeneous Treatment Effect Estimation Under Structure Uncertainty
- Proximal Causal Inference With Text Data
- Hyperparameter Tuning for Causal Inference with Double Machine Learning: A Simulation Study
- The Risks of Risk Assessment: Causal Blind Spots When Using Prediction Models for Treatment Decisions