Drift-Aware Off-Policy Evaluation for Retrospective Clinical Cohorts

Retrospective cohort studies judge treatment strategies by reweighting what clinicians actually did in the past. That reweighting depends on a model of how clinicians chose treatments, and clinical practice changes over the years a cohort covers. In the National Cancer Database breast cancer cohort, chemotherapy use among intermediate-risk patients fell from 25.8% in 2010 to 18.4% in 2013 (Chen et al.). Yet when machine learning researchers estimate the value of a new ventilation or sepsis policy from intensive care records, they fit one pooled model of clinician behavior to more than a decade of data. If practice drifted, the weights are wrong period by period, and the estimated benefit of the new policy can be biased in either direction.

We propose to fix this with an estimator that sweeps through the cohort in calendar order and keeps two learning processes running at different speeds. A fast process continually re-estimates the current clinician treatment policy. A slow process accumulates the long-run value of a fixed candidate policy, using importance weights taken from the fast process's current estimate. This is two-timescale stochastic approximation: two coupled iterative updates, where the slow one sees the fast one as already settled and the fast one sees the slow one as frozen. The new question is how fast "fast" should be. Track practice too slowly and the weights lag behind the drift. Track it too quickly and the weights become noisy. We aim to derive this trade-off, turn it into a rule for setting the step size, and test it on intensive care cohorts where off-policy evaluation numbers are already reported.

Core Claim

Off-policy evaluation (estimating how a policy would have performed using only data generated by a different policy) on retrospective cohorts carries a bias that has not been examined: drift in the behavior policy across calendar time. Our first claim is that a calendar-ordered, two-timescale estimator, with a constant-step behavior tracker coupled to gradient-corrected temporal-difference learning, removes most of this bias. Our second claim is that the best tracker step size can be predicted from an estimate of the drift rate, rather than tuned against a ground truth that real cohorts never provide.

The method is a standard tool applied in a new setting, so we want to be explicit about what is new. The theory of two-timescale updates under Markovian noise assumes the fast process tracks a target that is a fixed function of the slow variable (Convergence of Two-Timescale Markovian Stochastic Approximations). Here the fast target also moves on its own with calendar time. Neither that paper nor the non-asymptotic TDC analysis (Non-Asymptotic Analysis for Two Time-scale TDC) covers this case, and neither touches clinical data.

Research Program

Estimator. Let $k$ index treatment decisions sorted by calendar time, with state $s_k$, action $a_k$, reward $r_k$, and features $x_k = x(s_k)$. The fast process updates a behavior model $\mu_{\phi}$ (multinomial logistic, or a small network) by one stochastic gradient step on the log-likelihood of the observed action, with constant step size $\alpha$:

$$\phi_{k+1} = \phi_k + \alpha \, \nabla_\phi \log \mu_{\phi_k}(a_k \mid s_k).$$

The importance ratio $\rho_k = \pi(a_k\mid s_k)/\mu_{\phi_k}(a_k\mid s_k)$, where $\pi$ is the candidate policy, then feeds off-policy TDC($\lambda$). TDC($\lambda$) has an auxiliary vector $\omega$ updated with step $\gamma_k$ and value weights $\theta$ updated with step $\beta_k$, where $\beta_k/\gamma_k \to 0$ and $\beta_k \to 0$. Eligibility traces $e_k$ propagate credit across long ICU stays. We use the update exactly as analysed in Convergence of Two-Timescale Markovian Stochastic Approximations, substituting the tracked ratio for a known one.

Because $\alpha$ is held constant, the tracker never freezes. Its effective memory is about $1/\alpha$ decisions, which is the property that lets it follow practice drift.

Theory. We will bound the asymptotic bias of $\theta$ in terms of the steady-state tracking error of $\phi$. Standard analysis of constant-step tracking splits that error into two parts. One is lag, roughly $\delta/\alpha$ for a target moving $\delta$ per step. The other is noise, roughly $\sigma\sqrt{\alpha}$ for gradient noise of scale $\sigma$. Balancing the two gives an optimal $\alpha^\star \propto (\delta/\sigma)^{2/3}$. We will estimate $\delta$ from the data by comparing behavior models fit on adjacent calendar blocks, which turns the trade-off into a step-size rule.

The proof will extend the running-max stability argument of Convergence of Two-Timescale Markovian Stochastic Approximations to a fast target that drifts with bounded variation. It will then carry the resulting perturbation through the slow TDC fixed point. Here the earlier paper supplies the method, and the extension to a drifting fast target is our contribution. We expect only a bias bound, not almost-sure convergence, because the constant fast step rules out exact convergence.

Data. We use three settings, chosen to cover the cases where truth is known and where it is not.

The first is DTR-Bench's OberstSepsisEnv (DTR-Bench). We generate synthetic "cohorts" from it whose behavior policy drifts in a controlled way, either a smooth threshold shift or a sudden protocol change. The candidate policy's true value is then available by simulation. We run this under DTR-Bench's corruption tiers p through p3, because that benchmark found that observation noise and missing values degrade RL methods monotonically.

The second is the MIMIC-III mechanical ventilation cohort of 19,780 stays used by DeepVent, with extraction through MIMIC-Extract.

The third is the MIMIC-IV sepsis cohort used by Shared-Benchmark Regime Decomposition, built with the MIMIC-IV pipeline.

De-identification in these databases means calendar order is only known coarsely. We treat this as a real constraint: drift can only be tracked between coarse time blocks, and we will report how much it limits the method.

Relation to work that motivates and constrains the design. SBRD shows that clinician decisions are non-stationary. Its regime-aware model predicts held-out actions at 96.8% accuracy against 80.1% for maximum-entropy inverse RL. However, its regimes are short-term operational switches such as escalation or workload, fit in a staged offline pipeline. We target slow calendar drift instead, and we use the behavior model as an input to off-policy evaluation rather than as the end product.

DeepVent (Towards Safe Mechanical Ventilation) and OGSRL (Offline Guarded Safe RL) both report large policy gains estimated under stationary assumptions. They motivate this work: we ask how much of those gains survive drift-aware evaluation.

The causal DTR work (Designing Optimal Dynamic Treatment Regimes) constrains our claims. It is a reminder that unmeasured confounding is a separate bias that drift tracking does not address. We will say so explicitly and not claim causal identification beyond measured covariates.

Baselines

The naive reference point is the approach used in practice: one behavior model fit on the pooled cohort, followed by off-policy TDC($\lambda$) or least-squares temporal difference (LSTD) learning with plug-in importance ratios. It uses the same features, the same candidate policies, and the same data. It isolates exactly the cost of ignoring drift.

The strongest existing method is Fitted Q Evaluation (FQE), the evaluator DeepVent used. On MIMIC-III ventilation it reports a mean initial-state value of 0.743 for DeepVent, 0.572 for behavior cloning, and 0.502 for the physician policy (Towards Safe Mechanical Ventilation). Those are value estimates, not accuracy measurements. We found no reported result on Lacuna for off-policy evaluation error on MIMIC cohorts or on DTR-Bench under behavior drift, so FQE's accuracy must be measured here in the simulator. FQE is the strongest competitor because it uses no importance weights. In principle that makes it immune to errors in the behavior model, though it remains sensitive to distribution shift in the states visited. We will run FQE with the same function class as our value head and the same hyperparameter search budget. "Beating it" means lower mean absolute error to the simulated true value, averaged over drift scenarios, without worse error when there is no drift.

The ablation that isolates our key ingredient is a staged, calendar-aware estimator. It fits the behavior model offline with exponential time-weighting matched to our tracker's memory length $1/\alpha$, or with calendar-time splines and covariate interactions, and then runs LSTD. If this matches our coupled estimator, then the coupling itself contributes nothing and only the step-size rule remains as a contribution. A second ablation sets $\alpha$ by our drift-based rule versus by oracle tuning against simulated truth, to test whether the rule gives a near-optimal step size without the oracle.

Why not just…?

Why not just use FQE, which needs no behavior model at all? FQE avoids weighting, but under drift the states the candidate policy reaches are covered unevenly across eras, and FQE extrapolates into those poorly covered regions. DeepVent's own report that single-timescale DDQN value estimates exploded on out-of-distribution states shows this extrapolation can fail badly (Towards Safe Mechanical Ventilation). FQE is still a real threat in low-drift regimes, and our simulator sweep over drift magnitude will show where it is competitive.

Why not just add calendar time as a covariate to a pooled propensity model? This is the most serious alternative, and in its kernel-weighted form it is nearly equivalent to our fast tracker. We treat it as the main ablation above. Our expected advantage is narrower than "it fails". It needs a hand-chosen bandwidth or spline basis, and it handles interactions between drift and covariates (such as an age threshold that shifts over time) only if those interactions are specified. Our step-size rule picks the memory length from estimated drift. If the matched-bandwidth staged estimator ties ours, we will report the contribution as the theory and the bandwidth rule, not the coupled algorithm.

Why not just run LSTD, given that the feature dimension is small? For a fixed behavior model and linear features, LSTD reaches the same fixed point as TDC in closed form, and the $O(d^2)$ cost is trivial at 36 features. We include it in the staged ablation for exactly this reason. The streaming TDC form matters only when the importance weights change during the sweep. If time-blocked LSTD performs equally well, that result points practitioners to the simpler tool.

Proposed Experiments

The decisive check is in the simulator. In OberstSepsisEnv cohorts with injected drift, we plot the error of each estimator's value estimate against drift magnitude. Pooled-propensity TDC should degrade as drift grows. Our estimator, with its rule-chosen $\alpha$, should stay close to the oracle-tuned $\alpha$. Its error should follow the predicted U-shape in $\alpha$, with the minimum near $\alpha^\star$. If the U-shape is absent or the minimum sits far from the prediction, the theory is wrong.

On MIMIC-III ventilation, we re-evaluate the DeepVent, behavior-cloning, and physician policies under drift-aware weighting and report whether the 0.743 versus 0.502 gap shrinks, holds, or grows. On the MIMIC-IV sepsis cohort, we report the estimated drift rate between calendar blocks, which by itself measures how non-stationary these databases are. We will also check whether drift-aware evaluation changes the ranking of the candidate policies, since a change in ranking is the clinically meaningful outcome.

Finally, we run the single-decision case on the NCDB-style setting, with a horizon of one, so the method reduces to tracked inverse-propensity weighting. This sanity check uses the documented chemotherapy trend. It connects the method to the reweighting that epidemiologists already use, as in the propensity-score design of Tewari et al..