Power grids stay stable by splitting control across timescales. Fast local controllers hold voltage and frequency steady second by second. A slow planner looks hours ahead and hands them targets, called setpoints, without micromanaging each device. Intensive-care treatment has the same two-speed structure. Doctors adjust vasopressors (drugs that raise blood pressure) and intravenous fluids hour by hour to keep blood pressure up. Whether the patient is actually recovering shows in blood tests that arrive only every 12–24 hours: lactate clearance, albumin drift, the neutrophil-to-lymphocyte ratio. Learned sepsis treatment policies ignore this split. They squeeze both kinds of signal into one decision every few hours. We propose a two-level policy learned from historical ICU records. The slow level reads biomarker trajectories every 12 hours and chooses a clinically meaningful setpoint: a target band for mean arterial pressure and a fluid budget for the next window. The fast level learns hourly dosing that tracks that setpoint.
This matters for a statistical reason as well as a clinical one. Learning a treatment policy from logged data, with no new experiments, is called offline reinforcement learning. Its sample cost can grow exponentially with the number of sequential decisions. Hourly policies are clinically better but have very long horizons. Splitting the problem gives each learner a short horizon. Unlike learned "latent skills", the interface between the levels is a quantity clinicians already argue about and can audit: the blood-pressure target and how much fluid to give.
Core Claim
A sepsis policy whose slow layer picks explicit, interpretable hemodynamic setpoints from biomarker trajectories, and whose fast layer tracks those setpoints hourly, will match or exceed the best flat hourly policy on offline value estimates. It will also have lower-variance evaluation and lower dependence on rare high-severity data. The advantage should come from the explicit setpoint interface, not merely from adding trajectory features or adding hierarchy.
The idea rests on three pieces of prior work. Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment compared flat policies at 1, 2, 4 and 8-hour steps on MIMIC-III. Finer steps won, but that paper never combined timescales, and its learned history summaries degraded at longer prediction horizons. This motivates our design and supplies our evaluation pipeline. What are the Statistical Limits of Offline RL with Linear Function Approximation? proves that offline learning can require samples growing like $(d/2)^H$ in feature dimension $d$ and horizon $H$. This constrains any flat hourly design over a multi-day stay. The grid side supplies the method. Robust MPC with data-driven demand forecasting for frequency regulation with heat pumps runs a daily planner, a 15-minute predictive controller, and a 0.5-second tracking loop. It separates the slow and fast layers through a physical buffer, a water tank whose fill level the slow layer manages. Our cumulative fluid balance plays that buffer's role. Hierarchical Control in Islanded DC Microgrids With Flexible Structures shows the general pattern of an upper layer that sets references for decentralized lower-layer trackers.
The closest machine-learning precedent is hierarchical offline RL with learned latent skills, as in Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills. There, the skill a high-level policy selects is an uninterpretable code learned from data. Our interface is fixed to clinically defined targets, and we test directly whether that choice matters.
Research Program
The data is the MIMIC-III v1.4 sepsis cohort, built following Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment (18,377 stays), with lab timing handled through MIMIC-Extract.
The fast layer acts hourly on vitals and recent doses. It chooses from the same binned fluid and vasopressor actions used by Deep Reinforcement Learning for Sepsis Treatment. Its objective is a tracking reward. For a 12-hour window $k$ with setpoint $g_k = (m_k, b_k)$, where $m_k$ is the lower bound of the target mean-arterial-pressure band and $b_k$ the window's fluid budget, the hourly reward is
$$r_t = -\max(0,\, m_k - \mathrm{MAP}_t) - \lambda \max\big(0,\, F_t - b_k\big),$$
where $F_t$ is fluid given so far in the window and $\lambda$ trades pressure deficit against budget overrun.
The fast learner is batch-constrained Q-learning. Its horizon is at most 12 steps, and it is trained on all hours of all stays. To make that possible, every logged window is relabeled with the setpoint the clinician's actions actually achieved: the realized pressure floor and the realized fluid total. This hindsight relabeling is what makes a goal-conditioned tracker learnable from data where nobody recorded an explicit target.
The slow layer acts every 12 hours, the cadence at which slow labs actually arrive. Its inputs are a trajectory state: levels, first differences and second differences of lactate, albumin, neutrophil-to-lymphocyte ratio, creatinine and SOFA (the standard organ-failure score), plus cumulative fluid balance. It selects among roughly a dozen discretized setpoints and is rewarded by the outcome-shaped signal of Deep Reinforcement Learning for Sepsis Treatment: SOFA and lactate change, plus terminal survival. Its horizon over a five-day stay is about 10 steps rather than 120.
The trajectory features follow the biomarker literature. Association between albumin changes and prognosis in older patients with acute myocardial infarction found nonlinear risk in the change of albumin rather than its admission level. Association Between High-Sensitivity C-Reactive Protein Trajectories and the Incidence of Metabolic Syndrome found that a falling inflammatory marker still carried excess risk. Both cohorts differ from ICU sepsis, so these papers suggest which features to try; they do not supply thresholds we import. We deliberately do not hard-code the albumin inflection points from the heart-attack cohort. Spline features are learned from data instead.
The final policy composes the two layers. Every 12 hours the slow layer chooses a setpoint, and the fast layer executes hourly. The result is an ordinary hourly action sequence, which can be scored exactly like a flat 1-hour policy.
Baselines
Clinician reference. The naive point of comparison is the logged clinician behavior. On the 1-hour test set of Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment, this scores 94.09 on per-horizon weighted importance sampling (PHWIS, an offline estimate of how a new policy would have fared, computed by reweighting logged trajectories).
Strongest flat policy. The strongest existing method is the flat kNN-behavior batch-constrained Q-learning policy from the same paper, with a recurrent history summary. Its 2-hour version reports PHWIS 96.26 on that 1-hour test set. We will rerun it in our pipeline at 1-hour and 2-hour steps. Both it and our method get the same cohort, splits, behavior-policy estimator, reward, and an equal hyperparameter search budget, measured in GPU hours and in number of configurations. We will report both our reproduction and the published 96.26. Beating it means two things: a higher PHWIS than our reproduction of the flat 2-hour policy, with bootstrap intervals over test patients that exclude zero; and a larger effective sample size in the importance weights.
Original flat policy. We also include the 4-hour dueling double deep Q-network of Deep Reinforcement Learning for Sepsis Treatment as the historical reference.
Ablations. Three ablations isolate the key ingredients. First, flat hourly batch-constrained Q-learning given exactly our trajectory features, which tests whether the features alone explain any gain. Second, the same two-level structure with a learned latent goal in place of the explicit setpoint, in the style of Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills, which tests whether interpretability costs or helps. Third, our hierarchy with the slow layer's trajectory inputs replaced by snapshots, meaning current lab levels only.
Why not just…?
Why not just add trajectory features to a flat hourly policy? This is the most serious threat. The first ablation is designed to settle it. If the feature-augmented flat policy matches the hierarchy on PHWIS and on importance-weight effective sample size, the hierarchy adds nothing beyond its interpretability, and we will report that. Our expectation of a gap rests on the horizon argument of What are the Statistical Limits of Offline RL with Linear Function Approximation?. A flat hourly learner must still propagate value across about 120 steps, and adding features increases $d$ without shortening $H$.
Why not just use a generic hierarchical offline RL method with learned skills? Learned skills can shorten the horizon just as well. If the latent-goal ablation wins, explicit setpoints are not buying performance. Our bet is that relabeling with realized pressure and fluid targets gives the fast layer dense, well-covered supervision. Latent goals inferred from noisy ICU data have no such anchor. Separately, a clinician can veto or audit a pressure target but not a latent code. We will report the performance gap, whichever direction it goes.
Why not just use better outcome prediction? Supervised models in MIMIC-Extract reach mortality AUROC of 0.85–0.90, but they predict outcomes without recommending actions. Their intervention AUPRC also stays below 0.50. We use prediction only as a prerequisite check on whether trajectories carry signal, not as a competitor for treatment choice.
Proposed Experiments
The first experiment is a cheap go/no-go check. In The association between the neutrophil-to-lymphocyte ratio and mortality in critical illness, the admission neutrophil-to-lymphocyte ratio lost all association with mortality in sepsis. In the general ICU population, adding it to the SAPS I severity score moved AUC only from 0.746 to 0.757. We will test whether the 12-hour trajectory features add 28-day mortality discrimination in MIMIC-III sepsis over a snapshot model of SOFA plus current hemodynamics. If they add nothing, the slow layer has no information to act on. In that case we would collapse the design to a tracking policy with a fixed setpoint and report that negative result.
The main experiment is the PHWIS comparison on the 1-hour test set described in the Baselines section. Alongside it we report fitted-Q evaluation, which estimates policy value with a learned value function and therefore depends on different assumptions than importance sampling, and the effective sample size of the importance weights.
Two further checks are decisive. The first is a severity-stratified analysis. Deep Reinforcement Learning for Sepsis Treatment found its flat policy least reliable at SOFA above 15. If decoupling helps where it should, the hierarchy's advantage in effective sample size should be largest in that stratum. The second is a setpoint-concordance analysis. Among held-out windows, patients whose clinicians happened to achieve the slow layer's recommended pressure floor and fluid budget should show lower mortality than discordant windows after adjustment for severity. This matched-observational test does not depend on importance weights. The hierarchy is supported only if it beats the reproduced flat 2-hour policy and the feature-augmented flat ablation, and if concordance is associated with better outcomes.
References
- Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment
- What are the Statistical Limits of Offline RL with Linear Function Approximation?
- Robust MPC with data-driven demand forecasting for frequency regulation with heat pumps
- Hierarchical Control in Islanded DC Microgrids With Flexible Structures
- Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills
- MIMIC-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC-III
- Deep Reinforcement Learning for Sepsis Treatment.
- Association between albumin changes and prognosis in older patients with acute myocardial infarction
- Association Between High-Sensitivity C-Reactive Protein Trajectories and the Incidence of Metabolic Syndrome:A Retrospective Cohort Study
- The association between the neutrophil-to-lymphocyte ratio and mortality in critical illness: an observational cohort study