Is Deep Reinforcement Learning Ready for Practical Applications in Healthcare? A Sensitivity Analysis of Duel-DDQN for Sepsis Treatment

Mingyu Lu, Zachary Shahn, Daby Sow, Finale Doshi-Velez, Li-Wei H. LehmanarXiv 2020 · arXiv

Abstract

Deep Reinforcement Learning (DRL) offers a potential path to optimizing sepsis treatment through offline learning from electronic health records. However, the transition from theoretical models to clinical practice is hindered by a lack of robustness. This paper presents a systematic sensitivity analysis of the Duel-DDQN algorithm, investigating how arbitrary design choices—such as time discretization, treatment history, and random seeds—impact clinical recommendations. Our findings reveal that minor implementation variations can lead to significantly different, and sometimes dangerously aggressive, treatment policies, highlighting the current limitations of DRL for high-stakes medical decision-making.

Introduction

In domains like robotics or gaming, Reinforcement Learning (RL) agents are evaluated through active interaction with their environment. In healthcare, this is impossible; testing unproven treatment strategies on live patients is unethical. Consequently, researchers rely on offline RL, training models on retrospective hospital data like the MIMIC-III database.

The primary challenge for clinical deployment is robustness. Building these models requires dozens of "arbitrary" decisions regarding state definition, reward functions, and temporal resolution. If a model’s clinical recommendations fluctuate significantly based on these minor implementation details, it cannot be trusted by clinicians. This study investigates the stability of a state-of-the-art DRL architecture, the Dueling Double Deep Q-Network (Duel-DDQN), in the context of sepsis management.

Methodology

The authors conducted a sensitivity analysis on a Duel-DDQN with Prioritized Experience Replay (PER). This architecture is designed to estimate the value of specific actions (IV fluids and vasopressors) independently of the patient's overall state. The analysis focused on five key axes of sensitivity:

  1. Treatment History: Inclusion vs. exclusion of prior medication data.
  2. Time Discretization: Sampling patient states in 1-hour vs. 4-hour intervals.
  3. Reward Formulation: Short-term mortality risk vs. long-term survival and organ function (SOFA scores).
  4. Neural Architecture: Comparing Long Short-Term Memory (LSTM) vs. Gated Recurrent Unit (GRU) encoders.
  5. Random Seeds: Assessing convergence stability across different initial weights.
Figure 1: Illustration of time discretization and event placement on the clinical timeline.

The study utilized the MIMIC-III (v1.4) database, focusing on a cohort of 7,956 patients. The action space was defined as a 5x5 grid representing quartiles of IV fluid and vasopressor dosages.

Key Insights and Results

The Danger of the Markov Assumption

A critical finding was the model's sensitivity to treatment history. Many RL models assume the current state is sufficient for decision-making (the Markov assumption). However, excluding history led the model to recommend aggressive, potentially toxic doses of vasopressors. Without context of prior administration, the agent default to "more is better," increasing recommended vasopressor dosages by up to 1.8x and fluids by up to 3.1x.

Figure 5: Heatmap showing the aggressive treatment policy recommended by the model when treatment history is excluded.

Temporal Sensitivity and Instability

Time discretization significantly influenced the model's "sense of urgency." Shifting from 1-hour to 4-hour bins increased the frequency of high-dose recommendations, suggesting the policy is tied as much to the clock as to patient physiology. Furthermore, the authors identified a "many optima" problem: different random seeds produced policies with nearly identical quality scores (Weighted Doubly Robust values) but wildly different clinical actions. For vasopressors, the standard deviation of recommendations across seeds reached 10.7%, a level of instability unacceptable for clinical use.

Over-Aggression in Low-Risk Patients

Subgroup analyses revealed that DRL agents were 5-7 times more likely than human physicians to prescribe vasopressors to low-risk patients (SOFA < 5). This suggests a fundamental failure in the model's ability to correctly calibrate treatment intensity to patient severity.

Conclusion

While DRL holds promise for personalized medicine, this study demonstrates that current models are alarmingly sensitive to design choices. The lack of a "ground truth" in retrospective data means we cannot verify if these aggressive strategies are beneficial or harmful. Future research must prioritize the development of physiologically faithful simulations, the identification of stable "policy fragments" where models agree, and more robust off-policy evaluation methods before DRL can be safely integrated into clinical workflows.

Summary updated

Abstract

The potential of Reinforcement Learning (RL) has been demonstrated through successful applications to games such as Go and Atari. However, while it is straightforward to evaluate the performance of an RL algorithm in a game setting by simply using it to play the game, evaluation is a major challenge in clinical settings where it could be unsafe to follow RL policies in practice. Thus, understanding sensitivity of RL policies to the host of decisions made during implementation is an important step toward building the type of trust in RL required for eventual clinical uptake. In this work, we perform a sensitivity analysis on a state-of-the-art RL algorithm (Dueling Double Deep Q-Networks)applied to hemodynamic stabilization treatment strategies for septic patients in the ICU. We consider sensitivity of learned policies to input features, embedding model architecture, time discretization, reward function, and random seeds. We find that varying these settings can significantly impact learned policies, which suggests a need for caution when interpreting RL agent output.
BibTeX
@misc{lu2020is,
  author = {Mingyu Lu and Zachary Shahn and Daby Sow and Finale Doshi-Velez and Li-Wei H. Lehman},
  title = {{Is Deep Reinforcement Learning Ready for Practical Applications in Healthcare? A Sensitivity Analysis of Duel-DDQN for Sepsis Treatment}},
  year = {2020},
  eprint = {2005.04301},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.04301},
}
Figure