Predicting the Need for Blood Transfusion in Intensive Care Units with Reinforcement Learning

Yuqing Wang, Yun Zhao, Linda R. Petzold

TL;DR

This paper introduces a reinforcement learning (RL) approach to predict the optimal timing for blood transfusions in Intensive Care Units (ICUs). By leveraging patient data and employing transfer learning, their system shows promise in improving clinical outcomes and potentially reducing mortality rates.

What Problem Does This Paper Address?

Critically ill patients in the ICU often require blood transfusions due to anemia or coagulopathy. However, these transfusions are not always beneficial and can even lead to complications and increased healthcare costs if administered inappropriately. Physicians make these decisions based on a complex interplay of factors, but the urgency of the ICU environment can limit their ability to consider all relevant patient information. This can lead to suboptimal transfusion strategies. Traditional approaches often frame this as a classification problem (will a transfusion be needed?), but the authors argue that it's a sequential decision-making problem where current decisions impact future states. They aim to build a decision support tool that can learn optimal, dynamic transfusion policies.

Methods

The core of the paper lies in applying an off-policy batch reinforcement learning (RL) algorithm called discretized Batch Constrained Q-learning (BCQ). This algorithm is well-suited for learning from historical, static datasets without further interaction with the environment.

Key methodological components include:

  • Patient State Representations: They explore three recurrent neural network architectures (RNN, ODE-RNN, and CDE) to encode complex, time-varying patient data into a meaningful state representation for the RL agent.
  • Reward Design: Two reward functions are proposed: one based on long-term survival (28-day mortality) and another incorporating short-term clinical status changes (using the Sequential Organ Failure Assessment - SOFA score). The latter aims to capture more immediate patient well-being.
  • Transfer Learning (TL): Recognizing that data can be scarce in specific clinical settings (like the UCSF dataset), they employ TL to transfer knowledge learned from a larger dataset (MIMIC-III) to improve performance on the smaller one. This involves Q-value transfer and weight transfer.

Datasets

The study utilizes two real-world critical care datasets:

  • MIMIC-III (v1.4): A large, publicly available dataset from over 58,000 hospital admissions. This serves as the "source domain" for transfer learning.
  • UCSF Dataset: A smaller dataset from the San Francisco General Hospital and Trauma Center, containing data from patients admitted to a Level I trauma center. This serves as the "target domain," particularly for demonstrating the benefits of transfer learning.

Key Insights and Ideas

  • Sequential Decision-Making is Crucial: Framing blood transfusion as a sequential decision problem, rather than a simple classification, is key to developing more effective, dynamic treatment strategies.
  • State Representation Matters: The choice of how to represent patient states significantly impacts RL policy learning. RNN and CDE architectures generally outperformed ODE-RNN in their experiments.
  • Short-Term Feedback is Valuable: The reward function that incorporates short-term clinical status (SOFA score) generally led to better policy learning than solely relying on long-term survival. This suggests that immediate patient well-being is a critical factor in optimizing transfusion decisions.
  • Transfer Learning Bridges Data Gaps: For data-scarce environments like the UCSF dataset, transfer learning from a larger dataset (MIMIC-III) is highly effective, significantly improving both accuracy and learning efficiency.
  • RL Can Complement Physician Decisions: The simulations suggest that RL-guided policies can lead to tangible improvements in patient outcomes, indicating its potential as a valuable decision support tool.

Key Results

  • MIMIC-III Performance: On the larger MIMIC-III dataset, the RL policies achieved comparable matching accuracy to hospital policies, demonstrating the feasibility of the approach.
  • UCSF Performance with TL: Transfer learning on the UCSF dataset led to significant improvements:
  • Up to 17.02% improvement in accuracy in matching hospital policies.
  • Up to 18.94% jump-start and 21.63% asymptotic performance improvement in Weighted Importance Sampling (WIS).
  • Simulated Clinical Impact: Policy simulations on the UCSF dataset suggest that the transferred RL policies could:
  • Reduce estimated 28-day mortality rate by 2.74%.
  • Decrease the acuity rate by 1.18%.

Limitations and Future Work

  • Data Scarcity: The UCSF dataset's small size presented a challenge for direct RL policy learning, highlighting the necessity of transfer learning.
  • Generalizability: While promising, the results are based on specific datasets and patient populations. Further validation on diverse cohorts is needed.
  • Action Space: The current action space is binary (transfuse or not). Future work could explore a continuous action space to optimize transfusion dosages.
  • Evaluation: While WIS and accuracy are used, exploring more robust evaluation methods like doubly robust evaluation could be beneficial.
  • Reward Design: Investigating inverse RL for more principled reward design is a potential avenue.
  • Clinical Integration: The authors envision the RL policy as an "auxiliary advice" for physicians, suggesting future work on how to best integrate such tools into clinical workflows.

Summary updated

Abstract

As critically ill patients frequently develop anemia or coagulopathy, transfusion of blood products is a frequent intervention in the Intensive Care Units (ICU). However, inappropriate transfusion decisions made by physicians are often associated with increased risk of complications and higher hospital costs. In this work, we aim to develop a decision support tool that uses available patient information for transfusion decision-making on three common blood products (red blood cells, platelets, and fresh frozen plasma). To this end, we adopt an off-policy batch reinforcement learning (RL) algorithm, namely, discretized Batch Constrained Q-learning, to determine the best action (transfusion or not) given observed patient trajectories. Simultaneously, we consider different state representation approaches and reward design mechanisms to evaluate their impacts on policy learning. Experiments are conducted on two real-world critical care datasets: the MIMIC-III and the UCSF. Results demonstrate that policy recommendations on transfusion achieved comparable matching against true hospital policies via accuracy and weighted importance sampling evaluations on the MIMIC-III dataset. Furthermore, a combination of transfer learning (TL) and RL on the data-scarce UCSF dataset can provide up to $17.02% improvement in terms of accuracy, and up to 18.94% and 21.63% improvement in jump-start and asymptotic performance in terms of weighted importance sampling averaged over three transfusion tasks. Finally, simulations on transfusion decisions suggest that the transferred RL policy could reduce patients' estimated 28-day mortality rate by 2.74% and decreased acuity rate by 1.18% on the UCSF dataset.
BibTeX
@inproceedings{wang2022predicting,
  author = {Yuqing Wang and Yun Zhao and Linda R. Petzold},
  title = {{Predicting the Need for Blood Transfusion in Intensive Care Units with Reinforcement Learning}},
  booktitle = {BCB},
  year = {2022},
  doi = {10.1145/3535508.3545523},
}
Figure