A policy learned from intensive-care records can pass every statistical safety test on the pooled data and still harm patients at a hospital it was never tested on. Safe policy improvement methods ask one question before a learned treatment policy is released: is its expected outcome, with probability at least $1-\delta$, no worse than current clinical practice? They answer it by treating every patient in the historical log as an independent draw from one population. Multicenter retrospective cohort studies make a different assumption. They treat the hospital as a unit of replication, because hospitals differ in case mix, protocols, and how clinicians decide, and these differences change how treatments relate to outcomes. This proposal brings that design choice from observational epidemiology into safe policy deployment. The certificate becomes a statement about a hospital: with confidence $1-\delta$, a newly drawn hospital has at least a $1-\varepsilon$ chance of seeing the candidate policy match or beat its own current practice.
The work matters because hospital-to-hospital variation is large and well documented. Leave-one-hospital-out studies on the multicenter eICU database find severe shift between sites (Looking for Out-of-Distribution Environments in Critical Care). Meanwhile, the sepsis policies that report dramatic benefits were built and evaluated inside a single hospital's records (Offline Guarded Safe Reinforcement Learning; Deep Reinforcement Learning for Sepsis Treatment). A certificate that averages over patients cannot tell a regulator or a hospital adopting the tool whether that hospital should expect improvement. We propose a certificate that answers this question directly, and a test of whether it keeps its stated failure rate.
Core Claim
Pooled high-confidence safety tests underestimate risk at new sites. The reason is that between-hospital variation in a policy's benefit does not shrink as the number of patients grows. It shrinks only as the number of hospitals grows, and even then only toward the spread of true site effects, never to zero. Suppose we separate each hospital's estimated benefit into two parts, the true site-level benefit and the within-site estimation noise, and then bound a lower quantile of the site-level benefit distribution. The result is a deployment rule whose failure rate on held-out hospitals matches its nominal level. Pooled rules exceed that level when heterogeneity is substantial.
Concretely, let $\Delta_h = J_h(\pi_e) - J_h(\pi_{b,h})$ be the improvement at hospital $h$. Here $J_h$ is the expected return (for example, survival-weighted organ-failure reward) under that hospital's patients and dynamics, $\pi_e$ is the candidate policy, and $\pi_{b,h}$ is hospital $h$'s own current practice. The baseline value $J_h(\pi_{b,h})$ is observed directly as the mean return in that hospital's log, so only the candidate's value needs off-policy evaluation, meaning estimation of a policy's value from data collected under a different policy. Treating hospitals as exchangeable draws from a distribution $F$, the deployment criterion is
$$ \Pr_{\text{data}}\Big( q_\varepsilon(F) \ge 0 \;\text{ whenever we deploy} \Big) \ge 1-\delta, $$
where $q_\varepsilon(F)$ is the $\varepsilon$-quantile of site-level improvement. Each hospital supplies a noisy estimate $\hat\Delta_h = \Delta_h + e_h$, where $e_h$ has a variance $s_h^2$ that we estimate with a within-hospital bootstrap. Under a normal random-effects model with mean $\mu$ and between-site standard deviation $\tau$, the quantity to lower-bound is $\mu - z_{1-\varepsilon}\tau$, where $z_{1-\varepsilon}$ is the standard normal quantile. Its confidence bound comes from a bootstrap that resamples hospitals. We also build a distribution-free variant from order statistics of the site estimates and use simulation to check whether it stays conservative.
Relation to Prior Work
The statistical machinery comes from high-confidence off-policy evaluation and improvement. High-Confidence Off-Policy Evaluation clips large importance weights so that concentration bounds become usable. High Confidence Policy Improvement splits data into a set for searching candidates and a set for one final safety test, and shows that bias-corrected bootstrap bounds need far less data than concentration inequalities. We reuse the split-then-test structure but split by hospital. Both papers assume one stationary data-generating process, and that assumption is what we relax. Towards Safe Policy Improvement for Non-Stationary MDPs is the closest analogue in spirit. It shows that ignoring drift over time pushes safety-test failure rates as high as 40%, and that modeling the drift restores them to at most 5%. We make the same move across sites instead of across time, and heterogeneity across sites has a different structure: sites are exchangeable, whereas drift is smooth.
The candidate policies themselves come from baseline-anchored offline reinforcement learning. Safe Policy Improvement with Baseline Bootstrapping and its estimated-baseline extension fall back to cloned clinician behavior wherever the data are thin. Their guarantees are within-distribution, and our certificate sits on top of them.
Two lines of work motivate the gap. Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluation found on MIMIC-III that a policy that looks beneficial on average can harm high-severity patients. It audits patient subgroups found by a tree. We audit hospitals, a grouping that is fixed in advance and is the unit at which deployment decisions are actually made. Act or Defer shows that error guarantees computed over the whole population collapse on rare strata, and it names exchangeability between hospitals as its own open problem.
Confounding constrains the proposal. Confounding-Robust Policy Improvement and Offline Policy Evaluation and Optimization Under Confounding bound policy value under a bounded amount of hidden confounding. The first is limited to one-step decisions. The second is validated only in simulation and gives no finite-sample confidence guarantee. We do not solve confounding. We assume no unmeasured confounding within each site and report how conclusions change under a one-step sensitivity adjustment in the style of Kallus and Zhou. The claim is only that site heterogeneity is a separate failure mode, and one that can be tested for directly.
Research Program
We build three things.
The first is a multi-site version of the Sepsis Simulator used by Keramati et al. and Kausik et al. Each simulated hospital draws its own perturbation of the transition parameters, its own patient-severity mix, and its own clinician behavior policy, with a heterogeneity scale $\kappa$ that we control. Because the simulator gives exact values of $\Delta_h$ at any fresh hospital, it is the ground truth for calibration.
The second is a sepsis cohort from eICU, extracted across its 208 hospitals (Federated Learning in Multi-Center Critical Care Research). It follows the MIMIC-III recipe of 4-hour windows and a 5×5 grid of fluid and vasopressor doses (Deep Reinforcement Learning for Sepsis Treatment). MIMIC-III serves as one additional external site (MIMIC-III).
The third is the certification procedure. Hospitals are split three ways: a search set where candidate policies are trained, a certification set where site-level estimates $\hat\Delta_h$ and $s_h$ are computed and the quantile bound is formed, and a held-out set used only to score the decision. Within each site we estimate $\hat\Delta_h$ with per-decision weighted importance sampling, the estimator the COBS study found best under mild mismatch between policies. Fitted Q evaluation, a regression-based alternative, serves as a check.
We also develop a per-site extension. Empirical-Bayes shrinkage pulls each small hospital's noisy estimate toward the fitted site distribution. This yields a per-hospital decision to deploy or to defer to current practice, and it tests whether pooling across sites rescues small hospitals that cannot certify alone. The eICU finding that local-only models underperform global ones motivates this component.
Candidate policies come from Soft-SPIBB trained on the search hospitals. To keep importance weights well behaved on real data, we also use a controlled family that mixes the cloned behavior policy with the SPIBB policy at mixing weights from 0.1 to 1.
Baselines
The naive reference points are "do nothing," which always keeps current practice, has zero failure rate by construction, and fixes the floor for benefit, and "deploy on point estimate," which releases the policy whenever pooled estimated improvement is positive. The second is roughly how the MIMIC sepsis results were reported. Offline Guarded Safe RL reports estimated mortality of 0.0138 against 0.0632 for standard care with no confidence certificate. We cite that number only as the kind of single-site claim our held-out-hospital check is meant to stress. We do not claim it is reproducible on eICU.
The strongest competitor is the pooled high-confidence test of High Confidence Policy Improvement: bias-corrected bootstrap lower bounds on the pooled patient-level improvement, with its train/test split. We also run the stricter clipped empirical-Bernstein variant from High-Confidence Off-Policy Evaluation, which reports a lower bound of 0.154 against a true mean of 0.191 on Mountain Car, compared with −5,831,000 for plain Chernoff–Hoeffding. No reported result was found on Lacuna for pooled high-confidence policy improvement on eICU or on a multi-site sepsis cohort, so we will produce that number ourselves.
Fairness rules apply to every method. All methods see identical search and certification hospitals, identical candidate policies, identical estimators, the same $\delta$, and the same bootstrap budget in resamples. Only the aggregation and the bound differ. Beating the pooled test means one of two things. On the simulator, held-out-hospital failure rate at or below $\varepsilon$ where the pooled test exceeds it, for heterogeneity $\kappa$ above a threshold we will identify. At low $\kappa$, a deployment rate within a few points of the pooled test. If our method only ever wins by refusing to deploy, that is a loss.
Two ablations isolate the key ingredient. The first drops the separation of noise from true heterogeneity and takes quantiles of raw $\hat\Delta_h$. We expect it to be over-conservative, especially with many small hospitals. The second drops the quantile target and bounds only the mean across hospitals. We expect it to have the same failure mode as the pooled test.
Why not just…?
Why not just pool all hospitals and run the standard bootstrap safety test? Because its confidence interval narrows at a rate of one over the square root of the number of patients, while the risk faced by a new hospital is governed by the spread of true site effects, which does not narrow at all. With thousands of patients, the pooled test can certify a policy whose benefit is negative at a sizable fraction of sites. This is the main threat, and it is settled empirically. If on the multi-site simulator the pooled test's held-out failure rate stays at or below $\varepsilon$ across realistic $\kappa$ fitted to eICU, the proposal's premise fails, and we will report that.
Why not just use a cluster bootstrap that resamples hospitals? A cluster bootstrap gives a valid interval for the average improvement across hospitals, which is the right fix for correlated patients but the wrong target. A positive mean with large between-site spread still implies many harmed sites. The mean-only ablation is exactly this method, and the simulator measures the gap between it and the quantile bound as a function of $\kappa$. If that gap is negligible at the heterogeneity levels estimated from eICU, the cluster bootstrap suffices and our contribution reduces to that diagnostic.
Why not just re-run the safety test locally at each new hospital before deployment? Most eICU hospitals are small, and the leave-one-hospital-out study found that models trained only on target-hospital data often did worse than globally trained ones. We expect local tests to refuse nearly everywhere. The per-site shrinkage extension is the direct comparison. We report the fraction of hospitals, by size tier, that each rule can certify at matched failure rate.
Why not just audit subgroups with GIOPE? GIOPE finds patient subgroups after the fact and bounds each one with Cantelli's inequality under a fixed patient distribution. It says nothing about a hospital absent from the data. We run it with hospital identity available as a splitting feature and check whether its leaves predict held-out-site failures. It is complementary, not a replacement.
Proposed Experiments
The decisive test is calibration on the multi-site simulator. We sweep heterogeneity $\kappa$, the number of certification hospitals (10 to 150), and patients per hospital. For each method we record the realized held-out failure rate, meaning the fraction of certified deployments for which a fresh hospital has true $\Delta_h<0$, compared against $\varepsilon$, together with the deployment rate and the average true improvement delivered. The predicted signature is a crossover. At $\kappa=0$ all certified methods are calibrated and ours pays a small cost in deployment rate. As $\kappa$ grows, pooled and mean-only rules exceed $\varepsilon$ while ours tracks it. The size of the crossover $\kappa$ relative to the heterogeneity estimated on eICU decides whether the method matters in practice.
On eICU we first estimate $\tau$ for each candidate policy. That estimate alone is a new empirical fact: how much the benefit of a learned sepsis policy varies across real hospitals. We then compare certification decisions against held-out-hospital estimates, restricted to mixing weights where held-out effective sample size is large enough for those estimates to be trustworthy. The prediction is that at mixing weights where the pooled test certifies but our bound does not, held-out hospitals show a larger share of negative estimated improvement than at mixing weights where both certify. Transfer from MIMIC-III to eICU adds one more stress case, a policy certified at one academic center and applied to a network of community hospitals.
A negative-control check guards against artifacts. Taking the candidate equal to the cloned pooled behavior should give $\hat\tau$ reflecting only differences between each site's practice and the pooled average, and it should never be certified as an improvement. A sensitivity analysis applies a one-step confounding adjustment at several levels of $\Gamma$ to see whether the ranking between methods survives modest hidden confounding.
References
- Looking for Out-of-Distribution Environments in Critical Care: A case study with the eICU Database
- Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies
- Deep Reinforcement Learning for Sepsis Treatment.
- High-Confidence Off-Policy Evaluation
- High Confidence Policy Improvement
- Towards Safe Policy Improvement for Non-Stationary MDPs
- Safe Policy Improvement with Baseline Bootstrapping
- Safe Policy Improvement with an Estimated Baseline Policy
- Identification of Subgroups With Similar Benefits in Off-Policy Policy Evaluation
- Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models
- Confounding-Robust Policy Improvement
- Offline Policy Evaluation and Optimization Under Confounding
- Federated Learning in Multi-Center Critical Care Research: A Systematic Case Study using the eICU Database
- MIMIC-III, a freely accessible critical care database
- Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning