Pessimism on the Hazard Ratio: Low-Data Treatment Policies from Censored Survival Data

When a hospital wants to learn a better treatment policy from its own records, it faces two problems that are usually solved separately. The first is that outcomes arrive as times to an event, such as death or recovery, and many records end before the event happens because the patient was discharged, transferred, or the study closed. Statisticians call this right-censoring. The second is that a policy learned from logs will happily recommend treatments the logs rarely contain, because the model's guesses there are unchecked. Offline reinforcement learning, which learns decision policies from fixed logs without trying anything new, handles the second problem with pessimism: it subtracts an uncertainty penalty from the estimated value of each action, so poorly supported actions look worse. Clinical offline learning in current use flattens survival into a 0/1 reward at a fixed horizon. It then applies a generic penalty that needs thousands of patients before it beats standard care. In the EpiCare benchmark, pessimistic agents needed more than 2,000 episodes to surpass standard of care and often fell below random under 256 (EpiCare).

We propose putting the pessimism on the one quantity that a proportional hazards model says actually separates actions. A Cox model, the standard regression for survival data, writes a patient's instantaneous risk as a shared baseline risk over time multiplied by $\exp(\text{score})$. The baseline cancels whenever two treatments are compared for the same patient, so their comparison depends only on a low-dimensional hazard ratio. Its uncertainty is estimated from the censoring-aware partial likelihood, the Cox model's likelihood, built from the patients still under observation at each event time. We therefore propose to penalize only that ratio. Our hypothesis is that this yields lower-bound survival estimates that are correct about censoring, less conservative than whole-value penalties, and useful at the hundreds-of-patients scale where current methods fail.

Core Claim

At stage $k$ of a treatment sequence, with patient history $x$ and a discrete action $a$, we model the hazard of the event from the start of that stage as

$$h_k(t \mid x,a) = h_{0k}(t)\,\exp\big(g_k(x) + \phi(x,a)^\top \theta_k\big).$$

Here $h_{0k}$ is an unspecified baseline hazard, $g_k$ is a flexible prognostic score (a neural network or boosted model), and $\phi(x,a)$ is a small feature map of action-by-covariate interactions with coefficient vector $\theta_k$. Only $\theta_k$ distinguishes actions.

We estimate $\theta_k$ by partial likelihood after cross-fitting $g_k$, the same orthogonalization idea used in treatment-effect estimation. The resulting information matrix $\hat I_k$ is assembled from the patients still at risk at each event time, so it automatically grows less certain wherever censoring has thinned the data. The pessimistic survival probability to a landmark $t^*$ is

$$S^{\text{pess}}_k(t^* \mid x,a) = \exp\!\Big(-\hat H^{+}_{0k}(t^*)\,\exp\big(\hat g_k(x) + \phi(x,a)^\top\hat\theta_k + \beta\sqrt{\phi(x,a)^\top \hat I_k^{-1}\phi(x,a)}\big)\Big).$$

In this expression, $\hat H^{+}_{0k}$ is an upper confidence version of the Breslow estimate of cumulative baseline hazard, and $\beta$ sets the confidence level. We then pick the action that maximizes this lower bound.

The structural point is that, within one stage, baseline uncertainty multiplies every action equally and cannot change the ranking. Only the hazard-ratio penalty does. Generic methods instead penalize the whole value, including the shared prognostic part. We expect that to be the main source of their over-conservatism and data hunger. Across stages, the baseline upper bound matters because it scales continuation values. That is where tail variance from shrinking risk sets enters, which is the censored analogue of the variance-aware penalties in Adaptive Pessimistic Value Iteration.

Relation to Prior Work

The closest existing method, dtrSurv, solves multi-stage survival treatment regimes by backward recursion with generalized random survival forests. It handles dependent censoring through redistribution-to-the-right and carries survival probabilities backward across stages. We reuse its backward recursion, but it has no lower-confidence penalty. Its authors also deliberately reject proportional hazards to avoid misspecification. We take the opposite bet in the small-sample regime, and we test that bet directly against misspecification.

SVL shows that value functions under sparse terminal rewards equal sums of survival probabilities, and it fits hazards by maximum likelihood with censored timeouts. That supports our choice of a likelihood-trained, non-bootstrapped survival target, but SVL has neither pessimism nor a covariate hazard-ratio structure, and it is goal-conditioned.

Three clinical offline-learning systems motivate the gap from the other side. DeepVent applies conservative Q-learning (CQL), which pushes down values of unseen actions, to ventilation in MIMIC-III using a binary 90-day reward plus APACHE II shaping. OGSRL adds out-of-distribution guards and physiological constraints, but scores policies by terminal mortality and SOFA transitions. The DTR reexamination paper shows that method rankings flip with the reward definition and that off-policy evaluation can rank random policies first. That last finding constrains our evaluation design: our main results must come from settings with known ground truth.

The theory papers supply the template for penalties. APVI proves that variance-aware penalties are instance-optimal in tabular MDPs with uncensored transitions. Risk-sensitive pessimism builds covariance-based ellipsoid penalties for linear MDPs. Our penalty is the partial-likelihood analogue of these, extended to censored outcomes.

Finally, Survival Instinct shows that pessimism plus length bias in the data drives agents to stay in well-covered states even with zero reward. In clinical logs, censored stays end early for reasons unrelated to death. If an algorithm treats truncation as termination, it can learn spurious preferences. A partial likelihood removes censored patients from the risk set instead of scoring them, so we predict our method is immune to this artifact while CQL is not.

Research Program

We will build a Python implementation of stage-wise partially linear Cox fitting with cross-fitted prognostic scores. The implementation will provide sandwich and model-based information matrices and an upper bound on the Breslow cumulative hazard. Backward recursion will carry pessimistic continuation survival from each stage to the previous one, following the probabilistic augmentation of dtrSurv. Dependent censoring that is driven by observed history will be handled by inverse-probability-of-censoring weights in the partial likelihood, and the weighted sandwich variance will feed the penalty.

On the analysis side, we will derive a finite-sample statement of the following form. With probability $1-\delta$, the chosen policy's true landmark survival is at least its pessimistic estimate. Its shortfall relative to the best well-covered policy then scales with $\sqrt{\phi^\top I^{-1}\phi}$ along that policy, which is a single-policy coverage condition expressed in risk-set-weighted information. This condition holds only when proportional hazards holds for the contrast term. We will therefore also build a diagnostic that tests time-varying contrast coefficients using Schoenfeld-residual trends. When proportionality fails, the diagnostic will fall back to stratified baselines by time window.

Baselines

The do-nothing reference is the behavior policy itself: standard of care in EpiCare, or the clinician actions in semi-synthetic MIMIC-III. Every method must at least match it. Any method that falls below it at some sample size counts as unsafe at that size.

Behavior cloning is the imitation reference. DeepVent reports a fitted-Q-evaluation initial-state value of 0.572 for it, against 0.743 for CQL and 0.502 for physicians.

The strongest pessimistic RL competitor is CQL with a terminal survival reward, run as in DeepVent with and without severity-score shaping. Its reported EpiCare result is that it needs more than 2,000 episodes to beat standard of care. EpiCare also reports that its direct-method evaluation error reached an RMSE of 106.4 on a 100-point scale, which is why we score policies by simulation truth rather than off-policy estimates. We also run IQL, the other pessimistic method in EpiCare.

The strongest survival-native competitor is dtrSurv. We found no reported result on Lacuna for its survival probability on ARIC beyond its qualitative claim of beating standard care. We therefore rerun it ourselves on identical data with its default forest settings, plus a matched tuning budget.

Three ablations isolate our ingredients. The first is plug-in Cox ($\beta=0$), which isolates pessimism. The second is pessimistic Cox with the penalty applied to the full linear predictor including $\hat g$, which isolates the claim that penalizing only the hazard ratio matters. The third is pessimistic Cox with censored stays treated as event-free terminations, which isolates the censoring handling.

All methods get identical stage definitions, covariates, and censoring indicators. Each gets the same number of hyperparameter configurations, selected on a held-out simulator seed. $\beta$ for our method is chosen the same way rather than on test truth.

Beating a baseline means higher true landmark survival and restricted mean survival time under the learned policy. The difference must be significant across at least 20 data draws, and the method must never fall below the behavior policy at any sample size.

Why not just…?

Why not just fit a Cox model with treatment interactions and take the best predicted action? This plug-in rule is the most serious competitor because it already has the structural sample efficiency. It is our $\beta=0$ ablation. Our expectation is that it wins when the behavior policy covers all actions well. It should lose sharply when some action is rare in some covariate region, which is exactly where the information-matrix penalty is large. If plug-in Cox matches us across overlap levels, pessimism adds nothing here and we will report that.

Why not just use CQL with a survival reward and severity shaping? The reexamination paper shows that shaping changes which method wins, so its gains need not reflect survival. EpiCare shows that CQL is data-hungry below 2,000 episodes. Our censoring stress test asks directly whether CQL's truncation handling produces survival-instinct behavior.

Why not just bootstrap dtrSurv's forests and act on a lower quantile? This is a real threat because it adds pessimism without the proportional hazards assumption. Bootstrap forests at a few hundred patients may give unstable quantiles in sparse leaves. The experiments settle it by running exactly this baseline across sample sizes and across proportional-hazards violations. If it matches our method at small sample sizes, the claimed benefit of hazard-ratio structure is refuted.

Proposed Experiments

The primary testbed is EpiCare, extended with administrative censoring and history-dependent dropout. Here the true value of any policy can be computed by simulation. We sweep dataset sizes of 256, 512, 1,000, 2,000, and 8,000 episodes. The decisive result is whether pessimistic Cox beats standard of care below the roughly 2,000-episode threshold that EpiCare reports for CQL, without ever dropping below it.

The second testbed is semi-synthetic MIMIC-III (MIMIC-III), using the real sepsis and ventilation covariates and the real clinician actions. Outcomes are simulated from known hazard models in three families: proportional, mildly time-varying, and crossing hazards. Overlap is controlled by subsampling rare actions.

A survival-instinct stress test makes discharge-censoring correlate with shorter, healthier stays. We then compare CQL against our method, including under zero reward as in the Survival Instinct paper.

Finally, we apply all methods to the 945-patient ARIC cohort used by dtrSurv. On that cohort we report only agreement, pessimistic lower bounds, and support diagnostics, not value claims, given the documented unreliability of off-policy evaluation.

Two outcomes would falsify the proposal. The first is if the full-predictor-penalty ablation performs as well as hazard-ratio-only pessimism. The second is if bootstrap forests match our method in the small-sample regime once proportionality is violated by the amounts typical of the crossing-hazard scenario.