Development of a Reinforcement Learning Algorithm to Optimize Corticosteroid Therapy in Critically Ill Patients with Sepsis

Răzvan Bologheanu, Lorenz Kapral, Daniel Laxar, Mathias Maleczek, Christoph Dibiasi, Sebastian Zeiner, Asan Agibetov, Ari Ercole, Patrick Thoral, Paul Elbers, Clemens Heitzinger, Oliver KimbergerJournal of Clinical Medicine 2023 · DOI

TL;DR

This paper introduces a reinforcement learning (RL) agent designed to optimize corticosteroid therapy for critically ill sepsis patients. The RL agent learned a more restrictive treatment policy than human clinicians, suggesting withholding corticosteroids more often, and this policy was associated with lower ICU mortality in a retrospective analysis.

What Problem Does This Paper Address?

Sepsis is a life-threatening condition with high morbidity and mortality. While corticosteroids have been explored as a treatment, their optimal use (indication, dose, timing) remains controversial, with conflicting evidence on their benefit and potential adverse effects. Current guidelines suggest their use in specific scenarios, but identifying which patients truly benefit is challenging. This ambiguity leads to variations in clinical practice. The paper aims to address this by developing an AI-driven approach to personalize corticosteroid therapy, moving towards a more data-driven, "precision medicine" strategy.

Methods

The core of this research is the development and application of a reinforcement learning algorithm. Specifically, an actor-critic RL algorithm was employed. This type of algorithm learns by interacting with an environment (in this case, patient data) and receiving rewards or penalties based on its actions. The algorithm was trained to minimize ICU mortality, using a Markov Decision Process framework.

Key methodological aspects include:

  • State Representation: The "state" of a patient was defined by a vector of 277 clinical parameters, updated every 24 hours. This was later expanded to 379 features to capture more nuanced information.
  • Action Space: The RL agent could choose from five discrete actions, representing different cumulative 24-hour doses of systemic corticosteroids (null, 1-100 mg, 101-200 mg, 201-300 mg, and >300 mg hydrocortisone).
  • Reward Signal: The primary reward signal was ICU mortality. The agent was trained to maximize survival by minimizing mortality.
  • Off-Policy Evaluation: To assess the RL agent's performance without directly deploying it in a live clinical setting, a high-confidence off-policy evaluation (HCOPE) method was used. This allowed for comparing the RL agent's learned policy against the actual clinician behavior (behavior policy) using historical data.
  • Explainability: To understand the RL agent's decisions, Layer-wise Relevance Propagation was used to identify the most important input features. A random forest model was also trained to simulate clinician decision-making, allowing for a comparison of feature importance between the AI and human approaches.

Datasets

The study utilized the AmsterdamUMCdb database, a comprehensive intensive care unit (ICU) database containing high-resolution clinical data from 23,106 ICU admissions of 20,109 patients between 2003 and 2016.

  • Sepsis Cohort: Patients were identified using the Sepsis-3 criteria, resulting in a cohort of 3051 ICU admissions from 2946 distinct patients.
  • Data Split: This sepsis cohort was randomly split into three subsets:
  • Training set (70%): 2,136 ICU admissions.
  • Validation set (20%): 610 ICU admissions.
  • Testing set (10%): 305 ICU admissions.

No new benchmarks were introduced; the study leveraged an existing, well-established ICU database.

Key Insights and Ideas

  • Restrictive Corticosteroid Policy: The RL agent learned a significantly more restrictive policy for corticosteroid administration compared to actual clinician behavior. This suggests that current clinical practice might be over-treating with corticosteroids in some sepsis patients.
  • Mortality Benefit of Individualized Therapy: The study's findings imply that an individualized, data-driven approach to corticosteroid therapy, as learned by the RL agent, could lead to improved patient outcomes, specifically reduced ICU mortality.
  • Explainability is Crucial: The use of techniques like Layer-wise Relevance Propagation and random forest modeling to understand the RL agent's decisions is vital. It demonstrates that the AI's recommendations are based on clinically plausible factors, not just a "black box" output.
  • Different Drivers for AI vs. Human Decisions: While both the RL agent and clinicians relied on vital signs and lab values, the relative importance of certain factors differed. For instance, refractory shock was a primary driver for clinicians, whereas the RL agent showed a more nuanced approach, less solely dependent on this single factor.

Key Results

  • Agreement with Clinicians: The RL agent's policy agreed with documented clinician treatment in 59% of data points in the testing dataset.
  • Restrictiveness: The RL agent suggested withholding corticosteroids in 62% of patient states, compared to 52% by physicians.
  • Mortality Reduction: When patients in the testing set received therapy identical to the RL agent's suggestion, mortality was lower (22.38%) compared to cases where the RL agent's and physician's decisions differed (28.33%). This held true for both withheld and prescribed corticosteroids.
  • Expected Reward: HCOPE demonstrated that the 95% lower bound of the expected reward for the RL agent's policy was higher than that of the clinician policy.
  • Feature Importance: Vital parameters (blood pressure, heart rate) and laboratory values (leukocyte count, glycemia) were key predictors for both the RL agent and the clinician model. However, vasopressor use and PEEP were more relevant for the clinician policy.

Limitations and Future Work

The authors acknowledge several limitations:

  • Single-Center Data: The model was trained and tested on data from a single medical center. External validation is crucial before widespread clinical adoption.
  • Data Anonymization: The removal of notes meant that the withdrawal of life-sustaining therapies could not be accounted for, which is a significant factor in ICU decision-making.
  • 24-Hour Time Step: The use of a 24-hour time step for modeling patient trajectories is an artificial construct that creates more data points than a clinician might have at any single moment.
  • Corticosteroid Definition: The action space was defined based on cumulative 24-hour hydrocortisone equivalent doses, which might not capture the full complexity of corticosteroid regimens.
  • Regulatory Hurdles: The authors note that the current state of AI in intensive care faces regulatory challenges and uncertainty, hindering routine use.

Future work should focus on:

  • External Validation: Testing the algorithm on diverse datasets from multiple centers.
  • Prospective Evaluation: Conducting controlled trials to confirm the observed mortality benefits in a real-world clinical setting.
  • Refining the Model: Exploring more granular time steps or different action spaces for corticosteroid therapy.
  • Broader Application: Investigating the potential of RL for optimizing other complex treatment strategies in critical care.

Summary updated

Abstract

Individualized use of corticosteroids in sepsis may result in a mortality benefit, but optimal treatment policy may be more restrictive than the routine clinical practice. Whilst external validation is needed, our study motivates a 'precision-medicine' approach to future prospective controlled trials and practice.
BibTeX
@article{bologheanu2023development,
  author = {Răzvan Bologheanu and Lorenz Kapral and Daniel Laxar and Mathias Maleczek and Christoph Dibiasi and Sebastian Zeiner and Asan Agibetov and Ari Ercole and Patrick Thoral and Paul Elbers and Clemens Heitzinger and Oliver Kimberger},
  title = {{Development of a Reinforcement Learning Algorithm to Optimize Corticosteroid Therapy in Critically Ill Patients with Sepsis}},
  journal = {Journal of Clinical Medicine},
  year = {2023},
  doi = {10.3390/jcm12041513},
}
Figure