Offline reinforcement learning learns a treatment policy (a rule mapping a patient's current state to a dose or setting) from historical hospital records, with no new experiments. Its most popular safety device is pessimism: the learned estimate of how good an action is gets pushed down whenever the records contain little evidence for that action. Retrospective hospital records, however, do not end only when patients die or recover. They also end when patients are discharged, transferred, or the study window closes, which is called censoring. Current medical offline RL treats every record ending as a final outcome. We argue that pessimism makes this worse, not neutral. A pessimistic learner gives low value to whatever lies past the end of the data, so a censored ending looks like a bad ending. The learner is then implicitly rewarded for imitating whatever kept patients in the record longest, and in an intensive care unit (ICU) the patients who stay longest are often the sickest.
This proposal tests that failure mode directly and then fixes it. The fix borrows two tools from survival analysis, the statistics of time until an event. The first is bootstrapping the value past a censored ending, rather than zeroing it. The second is reweighting patients by how likely they were to remain observed. In two sentences: pessimistic offline RL secretly converts "stayed in the dataset" into reward, and in clinical cohorts with discharge and dropout this teaches the wrong lesson. We measure that bias, remove it with a censoring-weighted pessimistic update, and verify the fix on benchmarks where the censored outcomes are actually known.
Core Claim
Survival Instinct in Offline Reinforcement Learning studied pessimistic algorithms (CQL, IQL, ATAC, PSPI) on robotics benchmarks and found that they act well even with zero or negative rewards. The reason is that episodes end on failure, so longer episodes mark good behavior, and pessimism turns episode length into an implicit reward. The effect disappeared when episode length was fixed (HalfCheetah). That paper motivates our work, but it never considers what happens when episode length is set by an observation process that has nothing to do with success.
Our claim is that in retrospective clinical cohorts, the same mechanism produces a censoring-induced length bias. Conservative Q-learning (CQL), the dominant medical method, lowers values for poorly supported actions and treats the end of a record as terminal. Combined, these push value toward actions correlated with long observed stays rather than with survival. We further claim the bias is removable by a conservative update in which only observed deaths are terminal. In that update, censored endings bootstrap from an estimated continuation value, and each transition is weighted by the inverse probability of still being observed. The goal is a policy whose estimated value targets restricted mean survival time (expected survival time up to a fixed horizon $\tau$), not record length.
Research Program
The work has three parts.
Diagnosis. We first show the bias exists on real data. We train CQL in the standard way on the MIMIC-III sepsis cohort (MIMIC-III), but with the reward set to zero. If the learned policy is still systematically non-random, and its preferences track ICU length of stay, then record length is acting as a hidden reward. This mirrors the zero-reward test of the survival-instinct paper, moved to a setting where length is driven by discharge rather than failure.
Method. We write the treatment problem in discrete time. The reward $r_t = 1$ for each period alive, with no discounting and a horizon $\tau$, so a policy's value equals its restricted mean survival time. We then fit the Q-function, which estimates the value of taking action $a_t$ in state $s_t$, by minimizing
$$\alpha\,\mathcal{R}_{\mathrm{CQL}}(Q) \;+\; \mathbb{E}_{\mathcal{D}}\Big[\,w_t\,\big(Q(s_t,a_t) - r_t - (1-d_t)\,\hat V(s_{t+1})\big)^2\Big].$$
The symbols are as follows. $\mathcal{R}_{\mathrm{CQL}}$ is the standard CQL penalty from Conservative Q-Learning, which pushes down values of actions favored by the current policy relative to actions in the data. $\alpha$ sets the strength of that penalty. The indicator $d_t$ equals 1 only for an observed death, so administrative ends, transfers, and discharges are not terminal. $\hat V(s_{t+1})$ is the target value of the next state; at a censored final record it is the value of the last observed state, still subject to the conservative penalty. $w_t = 1/\hat G(t \mid h_t)$ is an inverse-probability-of-censoring weight. Here $\hat G$ is the estimated probability of remaining uncensored through time $t$ given the patient history $h_t$, fit with a discrete-time hazard network.
The weights correct for informative censoring, for example a discharge that happens because a patient improved. They do so under the assumption, also made by dtrSurv, that censoring is independent of outcome given the recorded history. We will stabilize and truncate the weights, and report sensitivity to truncation.
Evaluation. Off-policy evaluation estimates how a new policy would have performed using only old data. In this setting it is known to be unreliable. Position: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination found that random policies often outranked learned ones under weighted importance sampling and doubly robust estimators. EpiCare reports a Direct Method error far exceeding the reward scale. We therefore do not rest conclusions on standard off-policy evaluation. Instead we build two settings where censored outcomes are known.
The first is EpiCare, a simulator, into which we inject censoring of three kinds: administrative, random, and dependent on health state. True policy value there is computable by rollout. The second is a censor-then-recover construction on MIMIC-III. MIMIC links to Social Security death records, so outcomes after ICU discharge are known. We artificially censor trajectories at ICU discharge, train every method on the censored data, and score the resulting policies against the uncensored 90-day survival labels. For this we use a censoring-doubly-robust estimate in the spirit of Benchmarking Observational Studies with Experimental Data under Right-Censoring. That paper showed that naive complete-case handling of censoring produced 90–100% false rejections where the doubly robust approach held 5%. Its tools supply our evaluation method; we extend them from a single treatment decision to sequential ones.
Baselines
Clinician reference. The do-nothing reference is the clinicians' own policy, estimated by behavior cloning (supervised imitation of recorded actions) on the same state features. A clinical policy that is not beaten on known-outcome restricted mean survival time would mean the learned policy adds nothing.
Standard CQL (strongest competitor). This is the de facto standard: CQL with a binary terminal survival reward and record ends treated as terminal. It is the approach of DeepVent, Kaushik et al. and the AID dexmedetomidine model. For reference, DeepVent reported a fitted-Q-evaluation initial-state value of 0.743 versus 0.502 for physicians and 0.572 for behavior cloning. That number comes from a model-based evaluator on a different reward, so it is not comparable to our metric. We rerun this baseline ourselves on identical cohorts, features, 4-hour steps, network sizes, and an equal hyperparameter search budget, including the same grid over $\alpha$. "Beating it" means higher true restricted mean survival time on EpiCare and higher censoring-doubly-robust 90-day survival on censor-then-recover MIMIC, by more than the seed-to-seed spread over at least five seeds.
Batch-constrained Q-learning. We include BCQ, which restricts the policy to actions resembling those in the data. We follow the 2-hour configuration of Sun et al., who reported a per-horizon weighted importance sampling score of 96.26 versus 94.09 for clinicians. This tests whether policy-side constraints suffer the same length bias as value-side penalties.
dtrSurv. dtrSurv is the closest survival-native method: backward induction with random survival forests under dependent censoring. It has no pessimism and few stages, and we found no reported result on Lacuna for dtrSurv on MIMIC or EpiCare. We run it with three coarse stages on the same cohorts.
Ablations. Two ablations isolate the key ingredients. The first is bootstrap only: censored ends are non-terminal, but $w_t = 1$. The second is weights only: inverse-probability weights are applied, but censored ends remain terminal.
Why not just…?
Why not just use the full 90-day mortality label, since MIMIC links to death records? On MIMIC this is a real competitor. We therefore include it as an oracle upper reference: standard CQL trained on uncensored labels. Our method succeeds if it closes most of the gap between censored-data CQL and this oracle. The point is that most retrospective cohorts lack such linkage, including many single-center registries like the hemodialysis cohort. Dropping patients without complete follow-up is complete-case analysis, which Demirel et al. show can badly distort conclusions.
Why not just use dtrSurv? It already handles dependent censoring and has convergence guarantees. However, it relies on forests over a handful of stages, while ICU decisions span dozens of steps over dozens of features. It also has no mechanism against value extrapolation to unsupported actions. If dtrSurv on three coarse stages matches our method on EpiCare, that is evidence that fine-grained timing does not matter, and we will report it as such.
Why not just bootstrap at timeouts, as standard RL does for time limits? This is the most serious threat. Under purely random censoring it may be nearly the whole fix. The bootstrap-only ablation tests exactly this. We expect weighting to matter only under state-dependent censoring, and EpiCare lets us vary the strength of that dependence. If bootstrap-only matches the full method at every censoring level, the contribution narrows to the diagnosis plus a one-line fix, and we will say so.
Proposed Experiments
Zero-reward diagnostic. On MIMIC-III sepsis we compare zero-reward CQL with a random policy. We measure the correlation between the learned action values and remaining ICU length of stay, and its correlation with 90-day survival. A length correlation that greatly exceeds the survival correlation confirms the hidden-reward mechanism.
Censoring-level sweep on EpiCare. We vary the censoring proportion from 0 to 60% and the strength of its dependence on hidden disease state. We expect standard CQL's true restricted mean survival time to degrade as censoring grows, while the censoring-aware variant should stay flat. If standard CQL does not degrade, the core claim is false.
Censor-then-recover on MIMIC-III. We rank all methods against known post-discharge outcomes. A secondary check asks whether standard off-policy estimators, applied to the censored data, reproduce that ranking. This directly measures how censoring corrupts the evaluation problem documented by the Oxford position paper.
Data-size sweep. Following EpiCare's finding that offline RL trails standard care below roughly 2,000 episodes, we repeat the sweep at 500, 2,000, and 8,000 trajectories. This tells us whether the correction helps in the small-cohort regime typical of real retrospective studies.
References
- Survival Instinct in Offline Reinforcement Learning
- MIMIC-III, a freely accessible critical care database
- Conservative Q-Learning for Offline Reinforcement Learning
- Multi-stage optimal dynamic treatment regimes for survival outcomes with dependent censoring
- Position: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination
- EpiCare: A Reinforcement Learning Benchmark for Dynamic Treatment Regimes
- Benchmarking Observational Studies with Experimental Data under Right-Censoring
- Towards Safe Mechanical Ventilation Treatment Using Deep Offline Reinforcement Learning
- A Conservative Q-Learning approach for handling distribution shift in sepsis treatment strategies
- Reinforcement learning model for optimizing dexmedetomidine dosing to prevent delirium in critically ill patients
- Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment
- Dialysis session timing and outcomes: mortality and hospitalization differences across morning, afternoon, and night shifts in hemodialysis patients