An Observational Causal Bootstrap for Confounder-Adjusted Treatment Effects

When researchers estimate a treatment effect from observational data, they adjust for confounders (variables that affect both who gets treated and the outcome) and then attach a confidence interval. That interval usually answers a question nobody asked. It describes how the estimate would vary if we drew a fresh population. The benchmarks that dominate causal machine learning, such as IHDP, ACIC and Twins, instead score the estimate against the average effect in the units actually observed. For that target, the randomness that matters is mostly which of those units happened to be treated. We propose a bootstrap built to match that target. It creates a synthetic copy of the observed units in which every unit has both potential outcomes: the outcome it showed, plus the missing one filled in by a covariate-conditioned model. It then redraws treatment assignments from the estimated propensity score (the modeled probability of treatment given covariates). On each redraw it refits the entire adjustment pipeline, nuisance models and tuning included.

The claim is that two errors in current practice push in opposite directions, and only this procedure corrects both. Standard intervals are too wide because they charge for effect variation across covariates, which is fixed once we condition on the observed units. They are also too narrow in small samples because they treat the fitted nuisance models as known. The procedure also yields a direct split of interval width into design uncertainty and sampling uncertainty. As far as this file shows, no confounder-adjusted method currently reports that split.

What prior work establishes and leaves open

The mechanism comes from A Causal Bootstrap. That paper separates sampling uncertainty from design uncertainty, builds a pseudo-population with imputed counterfactuals, and redraws treatment. It shows that the Neyman variance estimator covers the truth about 99% of the time at nominal 95% under heterogeneous effects, while its own procedure is close to nominal. It is developed for randomized designs. Its imputation pairs the k-th percentile of treated outcomes with the k-th percentile of control outcomes (rank matching, also called isotone coupling) and uses no covariates. We keep the pseudo-population idea and change two things. First, missing outcomes are imputed from outcome models conditioned on covariates, so rank coupling applies only to the leftover residuals. Second, assignments are redrawn from an estimated propensity rather than a known design. This makes the method usable under confounding. It also shrinks the conservatism, because the part of effect heterogeneity that covariates explain is no longer unidentified.

Three papers motivate the regime. Gaussian and Bootstrap Approximation for Matching-based Average Treatment Effect Estimators shows that naive resampling fails for nearest-neighbor matching and proves a multiplier bootstrap valid there. It reports no numerical experiments. Because our procedure redraws assignments over a fixed set of units, it never duplicates units. Duplicated units create the ties that break the naive bootstrap for matching, so we can test empirically whether this sidesteps the failure. Coordinated Double Machine Learning trains the two nuisance models jointly, with a penalty on the covariance of their residuals, and tunes the penalty weight by cross-validation. Double machine learning means fitting separate models for treatment and outcome given covariates, then combining their residuals so that errors in the two models largely cancel. Coordinated double machine learning reduces finite-sample bias but provides no uncertainty quantification. The data-dependent tuning is exactly the kind of choice an analytic standard error ignores and a full-pipeline refit captures. Unified Covariate Adjustment for Causal Inference likewise relies on asymptotic normality alone.

Two papers constrain the design. Stochastic Neural Networks for Causal Inference with Missing Confounders hides gestational age in Twins and shows that adjustment methods built on the no-unmeasured-confounding assumption can fail badly: GANITE's error reached nearly 0.42, against 0.0135 for CI-StoNet. Our bootstrap assumes no unmeasured confounding, so it should undercover in that setting. We treat this as a stated limit to verify, not something to hide. Bayesian Nonparametric Adjustment of Confounding warns that adjusting for variables that drive only treatment inflates variance and bias. Because our refit repeats any variable-selection step on every redraw, that cost shows up in the interval width rather than being hidden.

Core Claim

Fix the n observed units with covariates $x_i$. The target is the sample average treatment effect

$$\tau_S = \frac{1}{n}\sum_{i=1}^{n}\big(Y_i(1)-Y_i(0)\big),$$

where $Y_i(1)$ and $Y_i(0)$ are unit i's potential outcomes under treatment and control.

The estimator $\hat\tau$ can be augmented inverse-propensity weighting, double machine learning, its coordinated variant, or nearest-neighbor matching. Its error relative to $\tau_S$ comes almost entirely from assignment randomness and from finite-sample error in the fitted nuisance models. We build a pseudo-population in which unit i keeps its observed outcome. Its missing outcome is set to $\hat\mu_{1-w_i}(x_i)$ plus a coupled residual, where $\hat\mu_w$ is the cross-fitted outcome model for arm w and $w_i$ is the observed treatment. The unit's residual quantile within its own arm, scaled by an estimate of the conditional spread, is mapped to the same quantile of the other arm's residual distribution. A coupling parameter interpolates from independent residuals to full rank coupling, which is the worst case.

Each bootstrap draw does three things. It samples $W_i^* \sim \text{Bernoulli}(\hat e(x_i))$ with clipping, where $\hat e$ is the fitted propensity. It reveals the matching pseudo-outcomes. It reruns the full pipeline, including cross-fitting and hyperparameter search, to get $\hat\tau^*$. We then form a basic-bootstrap interval from the quantiles of $\hat\tau^*-\tau_S^{\text{pseudo}}$.

To target the population effect instead, we additionally resample units. Comparing the two runs splits the interval width into design and sampling components.

The prediction has two parts. On sample-effect targets, this interval reaches nominal coverage while being narrower than influence-function intervals whenever covariates explain much of the effect heterogeneity. On small samples with flexible tuned learners, it widens relative to any analytic shortcut that treats the nuisances as fixed.

Research Program

The work has three parts.

The first is the procedure and its implementation. This covers the coupling scheme, propensity clipping, and warm-started refits that keep neural nuisance models affordable at about 200 to 500 draws.

The second is a validity analysis. We will state conditions on outcome-model and propensity error under which the pseudo-population assignment distribution approximates the true one. We will also show that the bias from residual coupling is one-sided, which makes the interval conservative only on the unexplained part of the heterogeneity.

The third is an empirical audit of which intervals the field reports actually cover the quantity the benchmarks score. Separating the two forces is the central scientific product: removing explained heterogeneity narrows the interval, and propagating nuisance-fitting variability widens it.

Baselines

The do-nothing reference is the influence-function interval reported by augmented weighting and double machine learning. This is the estimate plus or minus 1.96 times the empirical standard deviation of the estimated influence function. It is what nearly every applied paper reports, and it targets the population effect. We run it on exactly the same cross-fitted nuisance models as our method. Beating it means equal or better coverage of $\tau_S$ with smaller median width on IHDP, ACIC and Twins.

The strongest existing competitor for the sample-effect target is an analytic conditional-variance estimator. It drops the explained-heterogeneity term and plugs in an estimate of the conditional variance of the influence function. It removes conservatism for free, so it is the method most likely to make ours unnecessary. It gets the same nuisance fits and the same tuning budget. Beating it means it undercovers $\tau_S$ in small-sample or tuned-pipeline settings while our method holds nominal coverage at comparable width. If it matches us everywhere, our refit contributes nothing, and we will say so.

We also run two resampling competitors at equal numbers of draws and refits. One is the Efron pairs bootstrap with full refit. The other, for the matching estimator, is the multiplier bootstrap of Gaussian and Bootstrap Approximation for Matching-based Average Treatment Effect Estimators, which we implement ourselves. No reported empirical result was found on Lacuna for that method on any dataset. The original A Causal Bootstrap serves as a covariate-free reference. Its reported number is the near-99% Neyman over-coverage in experiments, and the file reports no observational result for it.

Three ablations isolate the key ingredients. Freezing the nuisance models across draws isolates the refit. Swapping covariate-conditioned imputation for marginal rank coupling isolates the role of covariates. Sweeping the coupling parameter from independent to rank coupling shows how much width the unidentified residual correlation costs.

For reference point-estimate numbers, the file gives ATE error of 0.0709 for CI-StoNet versus 0.0875 for CEVAE on ACIC 2019 (Stochastic Neural Networks for Causal Inference with Missing Confounders). No reported result was found on Lacuna for interval coverage on IHDP, ACIC or Twins, which is itself the gap this audit fills.

Why not just…?

Why not just report the influence-function interval, as double machine learning already prescribes? It is valid for the population effect. The benchmarks, however, score the sample effect, and for that target it charges for heterogeneity that is fixed. This is the observational analogue of the 99% over-coverage in A Causal Bootstrap. In small samples it also ignores nuisance error, which Coordinated Double Machine Learning shows can bias double machine learning in finite samples. Our coverage-versus-width curves across sample sizes will show whether either failure is large enough to matter.

Why not just use the analytic conditional-variance correction? This is the real threat. If nuisance-fitting variability is negligible, it delivers the narrowing at no cost. Our counter-hypothesis is specific: on IHDP-scale data with tuned learners, and especially with the cross-validated penalty weight of coordinated double machine learning, the analytic correction undercovers. The frozen-nuisance ablation and the comparison across sample-size subsamples settle this directly.

Why not just run the ordinary pairs bootstrap with refits? It mixes sampling and design uncertainty, so it targets the population effect and inherits the conservatism. For matching it duplicates units and creates the ties that make naive resampling inconsistent, as shown in Gaussian and Bootstrap Approximation for Matching-based Average Treatment Effect Estimators. We run it at equal compute and expect it to be wider for the sample effect and miscalibrated for matching.

Why not just take Bayesian posterior intervals from BART or CI-StoNet? Posteriors depend on the prior and on correct structure. Bayesian Nonparametric Adjustment of Confounding reports erratic year-to-year behavior from alternative Bayesian adjusters, and CI-StoNet's guarantees require a correctly specified graph. We include BART credible intervals as a secondary comparison but do not claim to beat them a priori.

Proposed Experiments

All datasets are ones where every unit's effect is known. On IHDP we use its simulated outcome realizations and record coverage of each realization's $\tau_S$. On ACIC 2019 we use the 10 datasets used by CI-StoNet. Twins is the cleanest test: both twins' mortality is observed, so $\tau_S$ comes from real outcomes. We simulate covariate-dependent treatment assignment with a known propensity, so that the true design is available as an oracle. On the IBM benchmark, subsampled to sizes from 500 to 20,000, we trace how the design and sampling components scale with n.

Estimators are augmented weighting, double machine learning with gradient boosting and with neural nuisance models, coordinated double machine learning with cross-validated penalty weight, and nearest-neighbor matching. Metrics are empirical coverage of $\tau_S$ at 95%, median interval width relative to the influence-function interval, and the fraction of width attributed to design uncertainty.

Three checks are decisive. First, on Twins with the oracle propensity, our interval should cover at about 95% and be clearly narrower than the influence-function interval. Swapping in the estimated propensity should cost little coverage. If it costs a lot, propensity error dominates and the method needs a correction. Second, the analytic conditional-variance baseline should fall below nominal on small-n IHDP and on coordinated double machine learning, while our full refit does not. If it does not fall below nominal, the refit is unnecessary and the contribution shrinks to the audit. Third, in the Twins variant with gestational age hidden, every method in the comparison, ours included, should undercover. Showing this keeps the no-unmeasured-confounding limit explicit rather than implied away.