The Meta Distribution of Hospital-Level Treatment Outcomes

Multicenter retrospective cohort studies usually report one number: a pooled treatment effect or a meta-analytic average across hospitals. That number cannot answer the question a health system actually faces, which is: in what fraction of hospitals does a treated patient have at least an 80% chance of surviving, or of benefiting? Wireless engineers met the same problem years ago. A network's average coverage probability can look healthy while some locations sit in permanent dead zones. Their fix was the meta distribution: the distribution, across frozen network layouts, of each location's own success probability. They estimate it from a few moments of that per-location probability and then reconstruct the whole curve.

We propose bringing this tool into observational causal inference. The estimand is the meta distribution of site-level counterfactual success probabilities: for each threshold $\alpha$, the fraction of hospitals whose patients would reach success probability at least $\alpha$ under a given treatment. The method rests on one observation. Within a single hospital, the outcomes of two distinct patients play the role that two independent fading draws play at one fixed wireless location. Averaging products of their bias-corrected outcome scores gives an unbiased estimate of the hospital's squared success probability, and higher moments follow the same way. Because the scores come from doubly robust estimation, the moments stay valid under confounding by indication, provided one of the two adjustment models is correct. The moments are then turned into a full curve with the same Beta-moment and higher-order reconstructions used in wireless networks.

Core Claim

The stochastic geometry work shows what averages hide. Two cases illustrate it.

  • Meta Distribution of Downlink NOMA in Poisson Networks compares two schemes for letting users share a channel. Restricting sharing to users near the cell center lets 92.1% of weak users reach reliability at least 0.5. Applying it everywhere lets only 19.9% do so. The average metric alone would not show a gap of this size.
  • Reliability in Post-Disaster Networks finds that switching off base stations around a disaster zone moves the fraction of users above reliability 0.6 from 78% to about 97%.

These results motivate the proposal. They show that the second moment of a conditional success probability carries decision-relevant information that the first moment does not.

The tool these papers supply is the moment structure. The $b$-th moment $M_b=\mathbb{E}[P_c^b]$ of the conditional success probability is easier to obtain than the distribution itself. A two-moment Beta fit reconstructs the curve well in most regimes (The Meta Distribution of SINR in UAV-Assisted Cellular Networks). It needs higher-order corrections near abrupt regime changes (On the Uplink SINR Meta Distribution of UAV-assisted Wireless Networks).

The claim of this proposal is that the same moment strategy, applied to doubly robust pseudo-outcomes, gives a consistent and confounding-adjusted estimate of the across-hospital distribution of counterfactual success. The method separates true between-hospital variation from within-hospital sampling noise without assuming a normal or logit-normal shape for the hospital effects.

The closest causal work is Collaborative Heterogeneous Causal Inference Beyond Meta-analysis. It shows that classical meta-analysis breaks down when sites have disjoint patient populations. It fixes this by fitting the adjustment models (propensity and outcome) collaboratively across sites. However, it targets only a single scalar average effect. We reuse its collaborative fitting for our adjustment models. What we add is a different estimand, the whole distribution across sites, and a new estimator for it.

Research Program

Let site $j$ have patients $i=1,\dots,n_j$ with covariates $X_i$, treatment $W_i$, and binary outcome $Y_i$. The site-level counterfactual success probability under treatment is $\mu_j=\mathbb{E}[Y(1)\mid \text{site } j]$, averaged over that site's own case mix. With cross-fitted propensity $\hat e$ and outcome model $\hat m$, the standard augmented inverse-propensity pseudo-outcome is

$$\phi_i = \hat m(X_i,j) + \frac{W_i\,(Y_i-\hat m(X_i,j))}{\hat e(X_i,j)},$$

whose conditional mean given site $j$ is $\mu_j$ when either nuisance is correct.

Within a site, distinct patients' pseudo-outcomes are conditionally independent given the site and the held-out nuisance fits. This gives the $b$-th moment estimator as a within-site U-statistic averaged over sites:

$$\hat M_b = \sum_j \omega_j \binom{n_j}{b}^{-1}\sum_{i_1<\dots<i_b \in j}\ \prod_{r=1}^{b}\phi_{i_r},$$

where $\omega_j$ are site weights (uniform for "fraction of hospitals", or proportional to $n_j$ for "fraction of patients"). Squaring site means instead would give $\mu_j^2$ plus the sampling variance of the site mean. That inflation is exactly the error the naive histogram of site estimates makes.

The distribution of $\mu_j$ over sites, $\bar F(\alpha)=\Pr_j(\mu_j\ge\alpha)$, is reconstructed in three ways:

  • the two-moment Beta fit;
  • a Fourier–Jacobi expansion using $\hat M_1,\dots,\hat M_4$;
  • moment-constrained upper and lower bounds on $\bar F(\alpha)$. These bounds hold whatever the true shape, so they show how much the Beta answer depends on its shape assumption.

The same construction applies to the site-level effect $\tau_j=\mu_j^{(1)}-\mu_j^{(0)}$, after rescaling from $[-1,1]$ to $[0,1]$. It answers questions of the form "in what fraction of hospitals does treatment lower mortality by at least 5 points?"

The theory work has three parts:

  • Establish $\sqrt{J}$-type consistency of $\hat M_b$, where $J$ is the number of sites, under cross-fitting.
  • Show product-rate double robustness for $b\ge2$. We expect a remainder term involving the propensity error times the outcome-model error, but with a factor of $b$ that grows with order.
  • Characterize the variance blow-up from inverse weights in higher-order products. We expect this to restrict practical use to $b\le4$ and to require propensity trimming.

When sites cover disjoint parts of the patient population, the adjustment models are fit collaboratively in the manner of Collaborative Heterogeneous Causal Inference Beyond Meta-analysis. That paper notes that its density-based weights become unstable at eight or more covariate dimensions. We therefore use classification-based propensity models and report how results change with dimension.

Baselines

Naive reference point. Compute a per-site doubly robust estimate of $\mu_j$ and take the empirical distribution of those estimates across sites. This is what an analyst would do by default. It treats sampling noise as real between-site spread, so its error should be largest for small sites and tail thresholds.

Normal-random-effects meta-analysis. Estimate site effects locally, then use the DerSimonian–Laird mean and between-site variance with a normal prediction distribution. This is the field's standard synthesis. Collaborative Heterogeneous Causal Inference Beyond Meta-analysis reports qualitatively that meta-analysis error "spikes significantly" as sites diverge, but gives no tabulated numbers.

Logistic mixed model. Fit a logistic regression with random hospital intercepts and treatment slopes, adjusted for covariates. We take the implied distribution of $\mu_j$ by averaging predictions over each site's case mix.

Nonparametric empirical-Bayes deconvolution. Estimate the mixing distribution of $\mu_j$ by nonparametric maximum likelihood (a method that recovers the distribution of true site rates from noisy site estimates) applied to the per-site doubly robust estimates. We treat it as the strongest competitor on this question. Like our method, it separates noise from true spread without a parametric shape assumption.

Fair comparison. All methods receive the same cross-fitted adjustment models (gradient-boosted trees, which beat deep tabular models on eICU in XMI-ICU), the same folds, the same propensity trimming, and the same hyperparameter budget.

Ablations. These isolate the key ingredients:

  • squared site means in place of within-site pair products;
  • outcome-model-only scores in place of doubly robust scores;
  • Beta reconstruction versus fourth-order reconstruction.

What counts as beating a baseline. Our method must achieve lower integrated absolute error of $\bar F(\alpha)$ over $\alpha\in[0.5,0.99]$. It must also achieve lower error in the fraction of sites below a clinically chosen threshold, with paired bootstrap confidence intervals that exclude zero.

We found no reported result on Lacuna for any method estimating an across-site distribution of counterfactual outcomes on eICU or any other cohort. The baselines will therefore be run fresh rather than compared against published figures.

Why not just…?

Why not just fit a hierarchical model with random hospital effects? A mixed model does give a between-hospital distribution, but its shape is fixed by assumption, usually normal on the logit scale. It can describe a skewed distribution with a cluster of failing hospitals only by stretching the variance. It also relies on a correct outcome model, which Long-Term Survival in Men With High Grade Prostate Cancer shows is risky under strong confounding by indication. We plant bimodal and skewed site distributions in the semi-synthetic experiments, where the logistic mixed model is the baseline expected to miss the tail. If it matches us when the truth is unimodal, we will report that honestly.

Why not just run a random-effects meta-analysis with a prediction interval? That approach gives only a mean, a spread, and a normal curve. It also requires valid local effect estimates at every site, which fails when sites treat non-overlapping patient groups, as Collaborative Heterogeneous Causal Inference Beyond Meta-analysis argues. The under-coverage stress condition in our experiments tests this directly.

Why not just deconvolve per-site estimates with nonparametric empirical Bayes? This is a real threat. It may match our method when sites are large and their noise is close to Gaussian. Our expected advantages are narrower. First, the U-statistic estimator does not need a model for each site's noise, which is badly non-Gaussian for small sites with rare outcomes. Second, the moment-constrained bounds give honest uncertainty statements that deconvolution does not. We will compare the two head to head while varying site size and outcome rate. If deconvolution wins everywhere, the contribution reduces to the bounds and to the estimand itself, and we will say so.

Why not just use conformal prediction intervals? Conformal prediction covers individual patients' outcomes on average over the population, as in A Machine Learning Approach to Predicting Heterogeneous Lipid Responses to Statins. That paper reached nominal 90% coverage, but some of its targets had very low predictability (for example $R^2\approx0.040$ for one lipid fraction). Patient-level intervals answer a different question from "what fraction of hospitals are reliable," and they do not adjust for confounding.

Proposed Experiments

Semi-synthetic benchmark. We keep the real covariates and hospital labels of the eICU myocardial-infarction subcohort used by XMI-ICU: 26,218 stays, with a reported 12.0% mortality. We simulate treatment assignment that depends on severity covariates, and outcomes whose site-level probabilities $\mu_j$ are drawn from known distributions: Beta, bimodal, and a heavy lower tail. Because the truth is known, every method can be scored against the true $\bar F(\alpha)$.

Varied conditions. We vary four factors:

  • confounding strength;
  • site size, including subsampling hospitals down to 10–20 patients;
  • base rate, including rare outcomes near zero, where wireless work found Beta fits degrade;
  • covariate overlap across sites, including deliberately disjoint case mixes.

Decisive checks. Three results must hold:

  • The pair-product ablation must show that squared site means overstate between-site spread, by an amount that grows as sites shrink.
  • The doubly robust version must stay accurate when exactly one adjustment model is deliberately misspecified, and the outcome-model-only version must not.
  • The fourth-order reconstruction must beat the Beta fit in the rare-outcome tail, mirroring the low-altitude breakdown of Beta fits in On the Uplink SINR Meta Distribution.

Real-data check without ground truth. Within each eICU hospital, we split patients in half and estimate the meta distribution from one half. We then test whether it predicts the observed distribution of event counts across hospitals in the other half better than each baseline, measured by held-out log-likelihood of site counts.

Transfer check. We examine whether MIMIC-IV, a single center, falls where the eICU meta distribution places it. This bears on the drop from 92.0 to 80.0 AUC that XMI-ICU reports when moving between these databases.

What the results would mean. The method earns its place if it is consistently more accurate in the tails at small site sizes, and if its bounds contain the truth at nominal rates. If nonparametric empirical-Bayes deconvolution matches it on all of these checks, the estimand remains useful, but the estimator does not justify itself.