Which Clock Should a Clinical Treatment Policy Read?

Reinforcement learning (RL) agents for sepsis treatment learn a policy, a rule mapping the patient's current measurements to a dose. They typically learn one rule that ignores time. Acute care is not time-free, though. The right action for an elevated lactate in hour 4 of resuscitation differs from the right action in hour 44. RL theory says that when an episode has a limited length, the best policy should depend on where you are in that episode. The obvious fix is to give the agent the remaining time.

In retrospective hospital records, that fix is a trap. A patient's "remaining time" ends at death or discharge, so it is computed from the outcome. Feeding it to the agent leaks the answer. Elapsed time since sepsis onset is legitimate, but it is misaligned across patients: one patient's hour 20 is another's hour 5 physiologically. This proposal argues for a third clock: the patient's position along their own biomarker trajectory. Examples are whether lymphocytes are still falling, at their low point, or recovering, or whether albumin is sliding or rebuilding. The policy's time-dependence is indexed by this biological phase rather than by wall-clock hours.

Core Claim

The claim has two parts, and both are testable. The first is a negative claim. Making a clinical RL agent depend on time by giving it the remaining time, which is the standard time-awareness recipe of Time Limits in Reinforcement Learning that appends $T-t$ to the observation, inflates offline value estimates when the horizon is stochastic and set by the outcome. When that feature must instead be predicted at deployment, the agent loses most of its apparent gain.

The second is a positive claim. A phase index estimated causally from noisy, irregularly sampled biomarkers recovers most of the benefit of an oracle remaining-time policy without leaking the outcome. It also beats elapsed time, because it aligns patients who progress at different speeds.

What is new is not "add time to sepsis RL." What is new is three things together:

  • treating the choice of clock as the central design variable;
  • auditing outcome leakage through that clock;
  • using longitudinal biomarker kinetics as the clock itself.

Why the pieces fit

The finite-horizon literature supplies the method but assumes the horizon is known in advance:

All three index non-stationarity by a deterministic, observed $t$ with known $T$. The file flags this as an open limitation, and it is exactly what fails in an ICU.

The biomarker literature supplies the clock and shows that trajectory phase carries information snapshots do not:

These papers motivate the proposal but are purely correlational, and they collapse trajectories into fixed-date cutoffs. We use them only as evidence that phase is a meaningful latent variable, not as validated control targets.

The medical RL work constrains the proposal:

The DTR-Bench result is a real warning. It says that giving a network temporal history does not by itself help. We therefore do not propose a bigger sequence model. We propose an explicit, low-dimensional clock.

Research Program

Phase estimator. For each tracked biomarker (lactate, lymphocytes, albumin, creatinine as available), we fit a local-linear-trend Kalman filter run strictly forward in time. A Kalman filter is a recursive estimator of a hidden level and slope from noisy readings. It handles irregular sampling natively, because the uncertainty grows between draws. It also smooths the pulsatile and stress-induced spikes the biomarker literature documents.

The filtered level, slope, and slope uncertainty are mapped to a small ordered set of phases $h \in \{1,\dots,H\}$, for example "worsening," "trough," "recovering," and "stable." The mapping comes from a hidden semi-Markov model fit on training trajectories only. Because filtering uses no future data, the phase is available at decision time by construction.

Policy that depends on the clock. The action-value is represented across the clock index as a low-rank tensor, extending Tensor Low-Rank Approximation of Finite-Horizon Value Functions from tabular states to a learned state embedding:

$$Q(s,a,h) \approx \sum_{k=1}^{K} u_k(s)\, v_k(a)\, w_k(h),$$

where $s$ is the snapshot state, $a$ the dose action, $h$ the clock value, $K$ the rank, $u_k$ a small neural network, and $v_k, w_k$ learned vectors. This shares statistical strength across phases, which matters because sicker phases are data-scarce. The Raghu et al. policy diverged from clinicians above SOFA 15, the regime where data are thinnest.

The same architecture is trained with three interchangeable clocks:

  • elapsed time since onset;
  • oracle remaining time, computed from the realized episode end exactly as a naive EHR pipeline would;
  • the biomarker phase.

Holding the architecture fixed means differences are attributable to the clock.

Backward-in-phase training where legal. In small tabular simulators we additionally train with the backward ordering of Beyond Stationarity, ordering by phase. This is only exact when phases are visited monotonically. We measure how often relapses violate that ordering and report backward training only where violations are rare.

Baselines

Snapshot reference. The first reference point is a stationary dueling double deep Q-network on snapshot states, following Deep Reinforcement Learning for Sepsis Treatment. Alongside it we report clinician behavior: standard of care on MIMIC-III, and the behavior policy in simulators. Any method that does not beat the stationary network on true simulator return contributes nothing.

Strongest feature-engineering competitor. This is the same network given elapsed time plus the filtered biomarker levels and slopes as ordinary input features. It is the cheapest way to obtain everything we claim, so it is the baseline that matters most.

Recurrent competitor. This is a recurrent Q-network of the kind benchmarked in DTR-Bench.

Strongest published offline-safe method on MIMIC-III sepsis. This is GMB-CPO from Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies. It reports an estimated mortality of 0.0138 against standard of care at 0.0632, and a model concordance rate of 0.549. We wrap our clock-indexed value function inside the same guard and constraint set. This tests whether the clock adds to the safety machinery rather than replacing it. Their mortality estimate comes from a learned model, so it is not directly comparable to a different cohort definition. We will re-run their pipeline on our extraction rather than quote the figure as a target.

Simulator results. For DTR-Bench environments, no reported result was found on Lacuna for per-environment numerical rewards. The file reports only qualitative winners: double DQN with a dueling head and recurrent SAC on AhnChemo, C51 on OberstSepsis, and no positive reward on SimGlucose under PK/PD variance (patient-to-patient differences in drug absorption and response). We will re-run these algorithms ourselves.

Fair comparison and what counts as winning. All learned methods share:

  • the same patient splits and the same simulator seeds;
  • the same parameter budget (matched to within 10% by adjusting width and rank);
  • the same number of gradient steps;
  • the same hyperparameter search of 30 configurations, selected on a validation split.

Beating a baseline means a higher true return in simulators, with non-overlapping 95% bootstrap intervals over at least 10 seeds. On MIMIC-III it means higher model concordance and lower unsafe-state rate at equal estimated mortality.

Key ablation. The ablation that isolates our ingredient gives the phase to the snapshot network as a plain input feature instead of as the tensor index. This separates "phase is informative" from "indexing by phase is the right structure."

Why not just…?

Why not just append remaining time, as time-aware RL recommends? On retrospective data, remaining time is a function of when the patient died or left, so it encodes the outcome. Our leakage audit tests this directly in simulators. We train with realized remaining time, evaluate offline using the same leaked feature, and then deploy with remaining time replaced by its best causal prediction. The prediction says the offline estimate is inflated and the deployed return falls. If deployed return does not fall, the leakage concern is empirically minor, and we will say so.

Why not just append elapsed time and biomarker slopes to a standard network? This is the real threat. The file's evidence against it is thin. In the wireless task of Tensor Low-Rank Approximation of Finite-Horizon Value Functions, a deep finite-horizon Q-network reached return 1.38 against 1.36 for the low-rank method. So time-as-input works when data are plentiful. We expect the factorized structure to help only in data-scarce phases. We therefore report results stratified by phase occupancy and by severity. If the feature baseline matches us everywhere, the conclusion is that the clinically important content is the phase signal, not the architecture. That is still a publishable correction to existing sepsis RL practice, but a narrower one.

Why not just use a recurrent network that learns its own clock? DTR-Bench found recurrent variants did not consistently beat feedforward ones under PK/PD variance, noise, and missingness, exactly the conditions clinical data bring. Our recurrent baseline re-tests this under matched budgets. We also probe whether its hidden state linearly decodes our phase. If it decodes the phase well and still underperforms, that points to optimization rather than representation as the bottleneck.

Proposed Experiments

The decisive experiments run first in simulators, where the true return is computable.

OberstSepsisEnv. We use this small simulated sepsis model from DTR-Bench and add stochastic, physiology-dependent termination: an episode ends when the patient reaches a recovery or death state rather than at a fixed step. The exact optimal non-stationary policy can then be computed by dynamic programming. We measure regret of each clock relative to it.

GhaffariCancerEnv and AhnChemoEnv. In these chemotherapy and radiotherapy simulators, we track a simulated toxicity marker to define phase. We ask whether phase-indexed dosing preserves the marker's recovery better than elapsed-time dosing at equal tumor response.

SimGlucoseEnv. This glucose–insulin simulator is the stress test under PK/PD variance. Here the goal is simply a non-negative reward, which no published algorithm reached according to the file.

Leakage audit. The audit is the same in every simulator: the gap between offline-estimated and true deployed return, compared across the three clocks.

MIMIC-III. We rebuild the Raghu et al. cohort with 4-hour bins and add filtered biomarker phases. Evaluation reports:

  • model concordance;
  • unsafe-state rate;
  • estimated mortality under the Yan et al. model;
  • the dose-deviation-versus-mortality curve, stratified by phase.

The stratified curve tests a specific prediction. If phase matters, the mortality-minimizing dose should shift between "worsening" and "recovering" patients who share the same snapshot SOFA.

Pre-registered falsification checks. The claim fails if any of the following holds:

  • phase-indexed policies do not beat elapsed-time policies in simulators with heterogeneous progression speeds;
  • the remaining-time leakage gap is within seed noise;
  • the feature-input ablation matches the tensor-indexed model in every severity stratum.