An interpretable RL framework for pre-deployment modeling in ICU hypotension management

Kristine Zhang, Henry Wang, Jianzhun Du, Brian Chu, Aldo Arévalo, Ryan Kindle, Leo Anthony Celi, Finale Doshi-VelezNpj Digit. Medicine 2022 · DOI · OpenReview

Introduction

Managing hypotension (dangerously low blood pressure) in the Intensive Care Unit (ICU) is a high-stakes balancing act. Clinicians must frequently decide between administering intravenous fluids, starting vasopressors, or a combination of both. While Reinforcement Learning (RL) is theoretically well-suited for finding optimal sequences of actions to maximize patient survival, standard "black-box" models often lack the transparency required for clinical trust. This paper introduces a framework that transforms complex RL into a transparent tool for ICU hypotension management.

The Interpretability Gap

Standard RL models typically provide opaque advice for every minute of a patient's stay, making it difficult for clinicians to interrogate the underlying logic. This framework addresses the interpretability gap by distilling massive amounts of clinical data into just 19 "decision regions" (DRs)—specific contexts where clinicians typically disagree. By focusing on these "forks in the road," the model provides succinct, human-verifiable treatment recommendations that bridge the gap between complex data-driven insights and clinical intuition.

Methodology

The authors developed a "decision point pipeline" that moves away from deep learning in favor of structured, interpretable modeling.

The decision point pipeline: from kernel learning to policy identification.
  1. Feature Selection: A Random Forest classifier identified the 20 most important clinical variables (e.g., MAP, lactate levels, urine output) for predicting treatment.
  2. Kernel Learning: Kernel-based methods were applied to learn a similarity metric, effectively mapping which patient states are clinically similar.
  3. Decision Region (DR) Identification: Instead of modeling every time step, the framework identifies clinical contexts where similar patients receive very different treatments from different doctors.
  4. Trajectory Compression: Patient stays are simplified into sequences of these DRs, reducing over 100,000 data points into a manageable set of transitions.
  5. Policy Learning: Because the state space is reduced to only 19 regions, the authors could use simple Policy Iteration rather than deep RL, ensuring the model remains fully auditable.
  6. Off-Policy Evaluation (OPE): Weighted Importance Sampling (WIS) was used to estimate how well the learned policies would have performed compared to historical clinical practice.

Data and Evaluation

The study utilized the MIMIC-III (Medical Information Mart for Intensive Care) dataset, a gold-standard public database.

  • Scope: 15,653 unique ICU stays involving hypotension (Mean Arterial Pressure < 65 mmHg).
  • Data Points: 123 patient condition variables recorded in 1-hour intervals for the first 72 hours of admission.
  • Split: 75% for training, 25% for a held-out test cohort.

Key Insights

A central innovation of this work is the focus on clinician disagreement. By highlighting regions of high variability, the model focuses on the most contentious—and therefore most valuable—decision points.

Detail of policy learning showing transitions between decision regions (DRs) toward clinical outcomes.
  • Succinctness as a Feature: Distilling the ICU experience into 19 regions creates a "map" that a human can actually read, allowing doctors to see exactly what physiological markers define one region versus another.
  • Fluid Stewardship: The model often recommended more conservative fluid use than clinicians in certain regions, aligning with a growing clinical movement to avoid complications like pulmonary edema.
  • High Confidence Estimates: By focusing on common "forks in the road," the model achieves a much higher Effective Sample Size (ESS) in its evaluations than previous RL healthcare studies, increasing statistical confidence in its recommendations.

Results

In retrospective evaluation, the RL-derived policies achieved higher WIS scores (a proxy for survival probability) than current clinical practice. For example, the "Final Survival-based" policy scored -0.30 compared to the clinicians' -0.67 (where higher is better).

The model successfully identified that in "borderline" hypotension, clinicians often over-react with vasopressors, whereas the model suggests "no treatment" may be safer when other vitals (like lactate) are normal. Clinical validation through specific patient trajectories showed exactly where the model diverged from human choices and why the model's path likely led to a better outcome.

Conclusion and Future Work

While the study is based on historical data and uses simplified binary action categories, it demonstrates a powerful framework for "human-in-the-loop" clinical decision support. Future work will aim to refine action dosages and address reward sparsity. Ultimately, this framework provides a transparent foundation for refining treatment guidelines before they are ever deployed at the bedside.

Summary updated

Abstract

Computational methods from reinforcement learning have shown promise in inferring treatment strategies for hypotension management and other clinical decision-making challenges. Unfortunately, the resulting models are often difficult for clinicians to interpret, making clinical inspection and validation of these computationally derived strategies challenging in advance of deployment. In this work, we develop a general framework for identifying succinct sets of clinical contexts in which clinicians make very different treatment choices, tracing the effects of those choices, and inferring a set of recommendations for those specific contexts. By focusing on these few key decision points, our framework produces succinct, interpretable treatment strategies that can each be easily visualized and verified by clinical experts. This interrogation process allows clinicians to leverage the model’s use of historical data in tandem with their own expertise to determine which recommendations are worth investigating further e.g. at the bedside. We demonstrate the value of this approach via application to hypotension management in the ICU, an area with critical implications for patient outcomes that lacks data-driven individualized treatment strategies; that said, our framework has broad implications on how to use computational methods to assist with decision-making challenges on a wide range of clinical domains.
BibTeX
@article{zhang2022interpretable,
  author = {Kristine Zhang and Henry Wang and Jianzhun Du and Brian Chu and Aldo Arévalo and Ryan Kindle and Leo Anthony Celi and Finale Doshi-Velez},
  title = {{An interpretable RL framework for pre-deployment modeling in ICU hypotension management}},
  journal = {Npj Digit. Medicine},
  year = {2022},
  doi = {10.1038/s41746-022-00708-4},
}
Figure