Abstract
Deep Reinforcement Learning (DRL) offers a potential path to optimizing sepsis treatment through offline learning from electronic health records. However, the transition from theoretical models to clinical practice is hindered by a lack of robustness. This paper presents a systematic sensitivity analysis of the Duel-DDQN algorithm, investigating how arbitrary design choices—such as time discretization, treatment history, and random seeds—impact clinical recommendations. Our findings reveal that minor implementation variations can lead to significantly different, and sometimes dangerously aggressive, treatment policies, highlighting the current limitations of DRL for high-stakes medical decision-making.
Introduction
In domains like robotics or gaming, Reinforcement Learning (RL) agents are evaluated through active interaction with their environment. In healthcare, this is impossible; testing unproven treatment strategies on live patients is unethical. Consequently, researchers rely on offline RL, training models on retrospective hospital data like the MIMIC-III database.
The primary challenge for clinical deployment is robustness. Building these models requires dozens of "arbitrary" decisions regarding state definition, reward functions, and temporal resolution. If a model’s clinical recommendations fluctuate significantly based on these minor implementation details, it cannot be trusted by clinicians. This study investigates the stability of a state-of-the-art DRL architecture, the Dueling Double Deep Q-Network (Duel-DDQN), in the context of sepsis management.
Methodology
The authors conducted a sensitivity analysis on a Duel-DDQN with Prioritized Experience Replay (PER). This architecture is designed to estimate the value of specific actions (IV fluids and vasopressors) independently of the patient's overall state. The analysis focused on five key axes of sensitivity:
- Treatment History: Inclusion vs. exclusion of prior medication data.
- Time Discretization: Sampling patient states in 1-hour vs. 4-hour intervals.
- Reward Formulation: Short-term mortality risk vs. long-term survival and organ function (SOFA scores).
- Neural Architecture: Comparing Long Short-Term Memory (LSTM) vs. Gated Recurrent Unit (GRU) encoders.
- Random Seeds: Assessing convergence stability across different initial weights.
The study utilized the MIMIC-III (v1.4) database, focusing on a cohort of 7,956 patients. The action space was defined as a 5x5 grid representing quartiles of IV fluid and vasopressor dosages.
Key Insights and Results
The Danger of the Markov Assumption
A critical finding was the model's sensitivity to treatment history. Many RL models assume the current state is sufficient for decision-making (the Markov assumption). However, excluding history led the model to recommend aggressive, potentially toxic doses of vasopressors. Without context of prior administration, the agent default to "more is better," increasing recommended vasopressor dosages by up to 1.8x and fluids by up to 3.1x.
Temporal Sensitivity and Instability
Time discretization significantly influenced the model's "sense of urgency." Shifting from 1-hour to 4-hour bins increased the frequency of high-dose recommendations, suggesting the policy is tied as much to the clock as to patient physiology. Furthermore, the authors identified a "many optima" problem: different random seeds produced policies with nearly identical quality scores (Weighted Doubly Robust values) but wildly different clinical actions. For vasopressors, the standard deviation of recommendations across seeds reached 10.7%, a level of instability unacceptable for clinical use.
Over-Aggression in Low-Risk Patients
Subgroup analyses revealed that DRL agents were 5-7 times more likely than human physicians to prescribe vasopressors to low-risk patients (SOFA < 5). This suggests a fundamental failure in the model's ability to correctly calibrate treatment intensity to patient severity.
Conclusion
While DRL holds promise for personalized medicine, this study demonstrates that current models are alarmingly sensitive to design choices. The lack of a "ground truth" in retrospective data means we cannot verify if these aggressive strategies are beneficial or harmful. Future research must prioritize the development of physiologically faithful simulations, the identification of stable "policy fragments" where models agree, and more robust off-policy evaluation methods before DRL can be safely integrated into clinical workflows.
Summary updated