Every well-run retrospective cohort study begins with a question the analyst must answer before touching outcomes: is this cohort large enough, and balanced enough, to detect the effect we care about? For static comparisons such as surgery versus radiation, epidemiologists answer it with power calculations and propensity-score overlap checks. Machine learning papers that learn sequential treatment policies from the same kind of historical records skip this step. These are policies that adjust fluids, vasopressors or ventilator settings hour by hour, learned by offline reinforcement learning (RL), meaning RL from a fixed log of past decisions with no new experimentation. Such papers report a single estimated improvement over clinicians and never ask whether a cohort of about 19,000 ICU stays could, even in principle, support that claim for a policy that departs from what clinicians actually did.
This note proposes a way to answer that question. The theory of off-policy sample complexity (how many logged samples are needed to learn or evaluate a new policy to accuracy $\epsilon$) already contains the needed ingredient. The error of an offline estimate is governed by coverage: how well the historical data spans the situations the new policy would visit. We turn that ingredient into a data-dependent audit for a given cohort and a given candidate policy. The audit reports a lower confidence bound on the policy's value. It reports a minimum detectable improvement, the smallest gain over clinicians that this cohort could certify, which plays the role of a power calculation. It also reports a tipping-point confounding level, the strength of unmeasured bedside information at which the claimed gain disappears. We then optimize policies against the audit itself, and test on a simulator with known ground truth whether it is calibrated where the standard practice, bootstrapped fitted-Q evaluation, is overconfident.
Core Claim
For sequential treatment policies learned from retrospective cohorts, the right analogue of a power calculation is a coverage-weighted uncertainty term taken from offline linear-MDP analysis, combined with a sensitivity allowance for hidden confounding. This audit has three properties. First, it controls the rate of false "beats clinicians" claims at its nominal level on a confounded simulator. On the same simulator, bootstrapped fitted-Q intervals exceed that rate once the candidate policy leaves well-covered regions. Second, it correctly predicts how the minimum detectable improvement shrinks as the cohort is subsampled. Third, policies optimized to maximize the certified lower bound achieve higher true value than policies optimized for point estimates whenever coverage is poor.
The direction of theory supplies the mechanism, and the retrospective cohort supplies the regime and the evaluation discipline. We do not claim a new optimal rate.
Where the mechanism comes from
The sample-complexity literature has recently shown that policy optimization from logged data can match the $\mathcal{O}(\epsilon^{-2})$ rate of value-based methods. The cleanest offline statement is the primal-dual method of A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs. It proves that $n = \mathcal{O}((C^*)^2 d^3 \log(\cdot) / ((1-\gamma)^2\epsilon^2))$ samples suffice under partial coverage. Here $d$ is the feature dimension, $\gamma$ the discount factor, and $C^*$ a feature concentrability coefficient measuring how poorly the data covers the target policy's features. The same analysis also handles safety constraints.
Inverting that bound gives an accuracy $\epsilon(n) \propto C^* d^{3/2} / ((1-\gamma)\sqrt{n})$ for a fixed cohort size. That is exactly the shape a power calculation needs. It plays a method-supplying role for us.
Its limits constrain the design:
- It assumes known features and exact linearity, which real EHR features violate.
- Its worst-case constants would declare every realistic cohort underpowered.
We therefore use its structure but calibrate its scale.
Other results rule out simpler routes. On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Method removes unverifiable bounded-importance-weight assumptions, but it tracks state-action occupancy directly, which the authors note is intractable in high-dimensional state spaces. Doubly Robust Off-Policy Actor-Critic protects against misspecifying one of two estimated components, but pays a $\Theta(\epsilon^{-4})$ rate and runs three critics. Both push us toward a feature-covariance quantity that can be computed from a cohort in closed form.
Concretely, let $\phi(s,a) \in \mathbb{R}^d$ be a fixed feature map of a patient state $s$ and treatment $a$. Let $\Lambda_n = \sum_{i=1}^n \phi(s_i,a_i)\phi(s_i,a_i)^\top + \lambda I$ be the regularized feature covariance of the cohort. Let $d^\pi$ be the distribution of states and treatments that a candidate policy $\pi$ would visit. The audited lower bound is
$$ \underline{V}(\pi) = \hat{V}(\pi) - \beta \, \mathbb{E}_{(s,a)\sim d^\pi}\big[\lVert \phi(s,a)\rVert_{\Lambda_n^{-1}}\big] - \Delta_\Gamma(\pi), $$
where the terms are defined as follows:
- $\hat V(\pi)$ is a linear fitted-Q value estimate.
- The middle term is the coverage penalty. The norm $\lVert x \rVert_{\Lambda^{-1}} = \sqrt{x^\top \Lambda^{-1} x}$ is large exactly where the cohort has few similar patients, and $\beta$ is a scale constant.
- $\Delta_\Gamma(\pi)$ is the worst-case value loss when hidden factors may have tilted clinicians' choices by at most a factor $\Gamma$. This follows the sensitivity model of Offline Policy Evaluation and Optimization Under Confounding.
The minimum detectable improvement for policy $\pi$ is the value $\delta$ at which $\underline{V}(\pi) - \hat V(\pi_{\text{clin}}) = \delta$ would first become positive, treated as a function of $n$. The tipping point $\Gamma^\dagger(\pi)$ is the smallest $\Gamma$ that drives the certified gain to zero.
Research Program
The work has four parts.
Calibrating the penalty. The constant $\beta$ is set on the confounded Sepsis Simulator of Kausik et al. We sweep cohort size, hidden-confounder strength and feature misspecification, and choose the smallest $\beta$ that gives nominal coverage there. We then freeze it. Sensitivity of all real-data conclusions to a factor-of-two change in $\beta$ is reported.
Building features for real cohorts. For MIMIC-III (MIMIC-III, a freely accessible critical care database), the feature map is a representation learned on a disjoint patient split and then frozen. The audit therefore never reuses data for both representation and certification. We record which records come from the CareVue era and which from the MetaVision era. This lets us check whether coverage shifts across the software migration, a within-database version of the transportability concern raised by multicenter cohort studies.
Optimizing against the certificate. We run policy optimization against the certified bound, using natural policy gradient with a log-linear policy, the parameterization for which Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policies proves fast convergence. Clinical hard constraints enter as cost constraints in the constrained-MDP form of Hong & Tewari.
Re-auditing published claims. We apply the audit to published claims, asking whether the policies reported by DeepVent and GMB-CPO lie inside the region a cohort of their size can certify.
The epidemiological discipline is taken from studies such as the propensity-scored prostate cancer cohort, which matched 453 men on observed covariates before comparing survival. Two elements carry over to the sequential setting. The coverage term is the sequential analogue of checking propensity overlap. The tipping-point $\Gamma$ is the analogue of asking how strong an unmeasured confounder must be to erase the finding. Neither element appears in current clinical offline-RL papers.
Baselines
Clinician policy (do nothing). The naive reference point is the clinician behavior policy itself, estimated by behavior cloning. Any certified gain is measured against it. On the ventilation cohort, fitted-Q evaluation values the physician policy at 0.502 and behavior cloning at 0.572 (Towards Safe Mechanical Ventilation Treatment Using Deep Offline Reinforcement Learning).
DeepVent (conservative Q-learning with fitted-Q evaluation). This is the strongest existing policy-learning competitor on ventilation. DeepVent is conservative Q-learning, which penalizes the value of treatments rarely seen in the data, evaluated by fitted-Q evaluation, which estimates a new policy's value by regressing its Q-function on logged transitions. It reports a mean initial-state value of 0.743.
GMB-CPO. This is the strongest competitor on sepsis, from Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies. It is model-based constrained policy optimization with a polynomial sum-of-squares support classifier acting as an out-of-distribution guardian. It reports an estimated mortality of 0.0138 against 0.0632 under standard care, a 78.2% reduction, with 0.549 concordance with clinicians.
Running the comparison fairly. DeepVent and GMB-CPO are re-run on the same patient splits, the same state and action definitions, and equal hyperparameter-search budgets. Every method's final policy is then scored by the same audit and, on the simulator, by true value.
Bootstrapped fitted-Q confidence intervals. This is the strongest competitor for the evaluation question, since it is what a careful practitioner would add today. It uses the same features and the same number of resamples.
Worst-case confounded evaluation. Kausik et al.'s worst-case evaluation is run at matched $\Gamma$.
What counts as beating them. On the simulator, we must meet nominal false-claim control, meaning the rate of certifying an improvement that does not exist. At the same time, our intervals must be narrower than the worst-case confounded bound, and our policies must have higher true value than point-estimate-optimized DeepVent-style policies in low-coverage settings. On MIMIC-III, where truth is unknown, we must predict the subsampling behaviour of estimates better than bootstrapped intervals do.
Ablations. Three ablations isolate the key ingredients: dropping the coverage term (keeping only $\Gamma$), dropping $\Gamma$ (keeping only coverage), and replacing the coverage term with a learned support classifier in the style of GMB-CPO's guardian.
Why not just…?
Why not just bootstrap fitted-Q evaluation? This is the real threat. If bootstrap intervals are already calibrated on retrospective cohorts, the coverage term adds nothing. Our expectation that they fail rests on one mechanism. Resampling the cohort reproduces sampling noise, but not the systematic extrapolation error that appears when a policy visits states with few similar patients. Every bootstrap replicate extrapolates the same way. DeepVent itself documents extrapolation failure: Double DQN overestimates value on out-of-distribution patients. Kausik et al. show that hidden confounders with memory add error that grows with the horizon, which no resampling scheme sees. The simulator experiment settles this directly. If bootstrap false-claim rates stay at nominal as policy coverage degrades, we will report that and recommend the bootstrap.
Why not just use conservative offline RL and trust its pessimism? Conservative Q-learning shrinks values of rare treatments, but its penalty has no statistical scale. It produces a better policy, not a statement of what the cohort can certify. Should I Run Offline Reinforcement Learning or Behavioral Cloning? shows offline RL beating cloning on noisy robot data: 92.4% versus 33.2% success on WidowX. It says nothing about calibration, safety or confounding. Our ablation that removes the coverage term while keeping optimization fixed isolates what the certificate itself contributes.
Why not just use propensity-score overlap diagnostics per decision? Checking overlap at each time step ignores compounding. A policy can overlap clinicians at every single step and still drive patients into physiological states the cohort never contains. That failure is exactly what Yan et al. identified. The occupancy-weighted coverage term measures overlap along the whole trajectory the new policy would generate. Comparing per-step overlap flags against our term on the simulator shows which one predicts large evaluation error.
Proposed Experiments
Simulator. The first decisive check is on the confounded Sepsis Simulator, where diabetes status acts as a persistent hidden confounder. We generate cohorts from 500 to 20,000 trajectories at several confounder strengths and ask three questions:
- Does the audit hold false improvement claims at nominal level while bootstrapped fitted-Q evaluation does not?
- Does the predicted minimum detectable improvement track the true one as $n$ changes?
- Do bound-optimized policies achieve higher true value than point-estimate-optimized ones when coverage is low?
Kausik et al. report their simulator results only as curves; no reported numeric result was found on Lacuna for the Sepsis Simulator. All simulator comparisons will therefore be re-run rather than quoted.
MIMIC-III. The second check uses the 18,923-patient sepsis cohort and the 19,780-patient ventilation cohort. We subsample each cohort, predict how the certified gain and its interval should change with $n$, and compare the prediction with the observed spread of estimates across subsamples.
Re-audit and headline results. We re-audit the DeepVent and GMB-CPO policies, reporting their certified lower bounds and tipping-point $\Gamma^\dagger$ values. The headline outcomes are either of two findings:
- The published improvements are not certifiable at the cohort sizes used. That would be a cautionary but useful result for clinical offline RL.
- They survive only up to a stated confounding strength, which is the first quantitative robustness statement for such claims.
The proposal fails if the calibrated $\beta$ from the simulator gives uninformative, near-infinite bounds on MIMIC-III, or if bootstrap intervals prove equally calibrated. Either outcome would be reported as the main result.
References
- A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs
- On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Method
- Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
- Offline Policy Evaluation and Optimization Under Confounding
- MIMIC-III, a freely accessible critical care database
- Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policies
- Long-Term Survival in Men With High Grade Prostate Cancer: A Comparison Between Conservative Treatment, Radiation Therapy and Radical Prostatectomy—A Propensity Scoring Approach
- Towards Safe Mechanical Ventilation Treatment Using Deep Offline Reinforcement Learning
- Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies
- Should I Run Offline Reinforcement Learning or Behavioral Cloning?