Optimizing warfarin dosing for patients with atrial fibrillation using machine learning

Jeremy Petch, Walter Nelson, Mary Wu, Marzyeh Ghassemi, Alexander P. Benz, Mehdi Fatemi, Shuang Di, Anthony Carnicelli, Christopher B. Granger, Robert P. Giugliano, Hwanhee Hong, Manesh R. Patel, Lars Wallentin, John W. Eikelboom, Stuart J. ConnollyScientific 2024 · DOI

TL;DR

Researchers developed a deep reinforcement learning (RL) model to optimize warfarin dosing for patients with atrial fibrillation, using a semi-Markov Decision Process to handle irregular clinical visits. The model, trained on over 22,000 patients, performed as well as expert-designed rules, with higher adherence to the model's suggestions correlating to significantly better patient outcomes and fewer strokes or bleeds.

What Problem Does This Paper Address?

Warfarin is a life-saving anticoagulant used to prevent strokes, but it is notoriously difficult to manage. It has a "narrow therapeutic window," meaning the dose must be precise: too little, and the patient risks a stroke; too much, and they risk a life-threatening hemorrhage.

Clinicians monitor this balance using the International Normalized Ratio (INR), aiming for a target range of 2.0–3.0. However, because warfarin’s effects are influenced by diet, genetics, and other medications, many patients spend significant time outside this safe zone. While simple "rules-based" algorithms exist, they are often "one-size-fits-all." This paper explores whether a data-driven, machine-learning approach can provide more personalized, dynamic dosing support.

Methods

The authors employed Batch-Constrained deep Q-learning (BCQ), a specific type of "offline" reinforcement learning. Unlike standard RL, which learns by trial and error in real-time, BCQ is designed to learn from historical datasets where the "agent" cannot interact with the environment. This is crucial for medicine, where you cannot ethically "experiment" with different doses on live patients.

Key technical innovations included:

  • Semi-Markov Decision Process (sMDP): Standard models assume actions happen at fixed intervals (e.g., every day). In reality, patients visit clinics at irregular intervals. The sMDP framework allowed the model to account for these varying time gaps.
  • Reward Function: The model was "rewarded" based on the time a patient spent in the 2.0–3.0 INR range, calculated using Rosendaal’s linear interpolation to estimate INR values between actual blood tests.
  • Feature Comparison: They built two versions—one using only INR and previous doses, and a "personalized" version that included clinical features like age, weight, and concurrent medications.

Datasets

The study leveraged an exceptionally high-quality dataset from four major international randomized clinical trials (RCTs):

  • Training/Tuning: Data from the ARISTOTLE, ENGAGE AF-TIMI 48, and ROCKET AF trials (22,502 patients).
  • External Validation: Data from the RE-LY trial (5,730 patients).

Using RCT data is a significant advantage over typical electronic health records because the data collection is rigorous, monitored, and covers patients across six continents, ensuring high "data hygiene" and global relevance.

Key Insights and Ideas

  • Simplicity vs. Complexity: Surprisingly, the "personalized" model (including age, weight, etc.) did not outperform the simpler model that focused only on INR and dose history. The authors suggest the complex model likely overfit to spurious correlations in the training data that didn't generalize to the validation set.
  • Aggressive Adjustments: The RL model tended to be more "aggressive" than human clinicians or the benchmark algorithm, recommending larger dose changes (over 20%) when a patient’s INR was significantly out of range.
  • The "Quality Care" Control: To prove the model's success wasn't just a proxy for "better hospitals," the researchers tested it against patients taking dabigatran (a drug that doesn't require dose adjustments). They found no correlation there, confirming that the benefits seen in the warfarin group were specifically due to better dosing, not just generally higher-quality hospital care.

Key Results

The study used "algorithm-consistent dosing" at the center level to measure success:

  • TTR Improvement: Every 10% increase in a center's consistency with the RL model predicted a 6.78% improvement in the patient's Time in Therapeutic Range (TTR).
  • Clinical Outcomes: A 10% increase in model consistency was associated with an 11% decrease in a composite outcome of stroke, systemic embolism, or major hemorrhage (Hazard Ratio 0.89).
  • Benchmarking: These results were comparable to a high-performing, expert-validated rules-based algorithm, proving that RL can reach "expert level" performance purely from data.

Limitations and Future Work

  • Retrospective Nature: The study looks backward at existing data. A prospective randomized trial is needed to see how the model performs when clinicians use it in real-time.
  • The "Black Box" Problem: Deep RL models are notoriously difficult to interpret. While the authors provide heatmaps of the policy, the underlying "why" of a specific dose recommendation remains opaque compared to simple rules.
  • Trial vs. Real World: Clinical trial patients are often more adherent and healthier than the general population. The model's performance might vary in "messier" real-world settings where patients miss doses or tests more frequently.

The authors have made their source code and a demonstration web tool available, signaling a move toward transparent, reproducible AI in clinical decision support.

Summary updated

Abstract

While novel oral anticoagulants are increasingly used to reduce risk of stroke in patients with atrial fibrillation, vitamin K antagonists such as warfarin continue to be used extensively for stroke prevention across the world. While effective in reducing the risk of strokes, the complex pharmacodynamics of warfarin make it difficult to use clinically, with many patients experiencing under- and/or over- anticoagulation. In this study we employed a novel implementation of deep reinforcement learning to provide clinical decision support to optimize time in therapeutic International Normalized Ratio (INR) range. We used a novel semi-Markov decision process formulation of the Batch-Constrained deep Q-learning algorithm to develop a reinforcement learning model to dynamically recommend optimal warfarin dosing to achieve INR of 2.0-3.0 for patients with atrial fibrillation. The model was developed using data from 22,502 patients in the warfarin treated groups of the pivotal randomized clinical trials of edoxaban (ENGAGE AF-TIMI 48), apixaban (ARISTOTLE) and rivaroxaban (ROCKET AF). The model was externally validated on data from 5730 warfarin-treated patients in a fourth trial of dabigatran (RE-LY) using multilevel regression models to estimate the relationship between center-level algorithm consistent dosing, time in therapeutic INR range (TTR), and a composite clinical outcome of stroke, systemic embolism or major hemorrhage. External validation showed a positive association between center-level algorithm-consistent dosing and TTR (R<sup>2</sup> = 0.56). Each 10% increase in algorithm-consistent dosing at the center level independently predicted a 6.78% improvement in TTR (95% CI 6.29, 7.28; p < 0.001) and a 11% decrease in the composite clinical outcome (HR 0.89; 95% CI 0.81, 1.00; p = 0.015). These results were comparable to those of a rules-based clinical algorithm used for benchmarking, for which each 10% increase in algorithm-consistent dosing independently predicted a 6.10% increase in TTR (95% CI 5.67, 6.54, p < 0.001) and a 10% decrease in the composite outcome (HR 0.90; 95% CI 0.83, 0.98, p = 0.018). Our findings suggest that a deep reinforcement learning algorithm can optimize time in therapeutic range for patients taking warfarin. A digital clinical decision support system to promote algorithm-consistent warfarin dosing could optimize time in therapeutic range and improve clinical outcomes in atrial fibrillation globally.
BibTeX
@inproceedings{petch2024optimizing,
  author = {Jeremy Petch and Walter Nelson and Mary Wu and Marzyeh Ghassemi and Alexander P. Benz and Mehdi Fatemi and Shuang Di and Anthony Carnicelli and Christopher B. Granger and Robert P. Giugliano and Hwanhee Hong and Manesh R. Patel and Lars Wallentin and John W. Eikelboom and Stuart J. Connolly},
  title = {{Optimizing warfarin dosing for patients with atrial fibrillation using machine learning}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-55110-9},
}
Figure