Abstract
This paper presents a deep reinforcement learning (DRL) framework to estimate optimal Dynamic Treatment Regimes (DTRs) from observational medical data. This framework is more flexible and adaptive for high-dimensional action and state spaces than existing reinforcement learning methods, aiming to model real-life complexity in heterogeneous disease progression and treatment choices. The proposed DRL framework comprises a supervised learning step to predict expert actions and a deep reinforcement learning step to estimate the long-term value function of DTRs. Implemented on a dataset from the Center for International Bone Marrow Transplant Research (CIBMTR) registry, the study focuses on prevention and treatments for graft versus host disease (GVHD) after transplantation. Experimental results demonstrate promising accuracy in predicting human experts' decisions and a high expected reward function for the DRL-based regimes.
Introduction
In the management of chronic and complex diseases, treatment is rarely a single event. Instead, it is a sequence of decisions adapted over time to a patient’s evolving clinical status, a concept known as Dynamic Treatment Regimes (DTRs). Optimizing these sequences is a cornerstone of precision medicine. However, traditional statistical methods often struggle with the high-dimensional and messy nature of real-world clinical data. For instance, in treating Graft Versus Host Disease (GVHD), clinicians may choose from hundreds of potential drug combinations. Existing models often require extensive manual feature engineering or oversimplify these complex action spaces, limiting their utility in actual clinical practice.
Methodology: A Two-Step DRL Framework
To address these challenges, the researchers developed a framework based on Deep Q-Networks (DQN) that operates in two distinct phases:
- Supervised Learning (The Screening Step): Given the vast action space (e.g., hundreds of drug combinations), a standard RL agent might explore medically unsound options. To prevent this, a feed-forward Deep Neural Network (DNN) is first trained to predict the actions taken by human experts. This supervised step acts as a filter, constraining the RL agent to consider only medically plausible treatment paths.
- Deep Reinforcement Learning (The Optimization Step): Using the screened actions, a DQN estimates the long-term value function. Unlike simple imitation, the DQN seeks to maximize a heuristic reward function representing patient quality of life (e.g., being alive, relapse-free, and GVHD-free). The model employs experience replay and target networks to stabilize learning from retrospective observational data.
Data and Experimental Setup
The study utilized longitudinal data from the Center for International Bone Marrow Transplant Research (CIBMTR) registry, focusing on 6,021 patients with Acute Myeloid Leukemia (AML) who underwent transplantation between 1995 and 2007. The action space included 127 drug combinations for GVHD prevention and 283 for acute GVHD treatment. Registry data was chosen over regional Electronic Health Records (EHR) due to its standardized, long-term follow-up across multiple facilities.
Results
The framework demonstrated a strong ability to capture clinical expertise and optimize outcomes. The supervised DNNs achieved high "Top-N" accuracy, indicating that deep learning can effectively model the patterns behind complex expert decision-making.
In terms of clinical outcomes, the DQN-based policy outperformed several benchmarks in simulations. For acute GVHD, the DQN achieved a mean reward of 0.717, compared to 0.673 for observed expert treatment and 0.677 for a Random Forest model. Notably, the model's prediction accuracy for chronic GVHD improved as the post-transplant period extended, reflecting the increasing stability of patient conditions over time.
Discussion and Conclusion
This research highlights the potential of medical registries as a "gold mine" for training DRL models, providing the longitudinal consistency necessary for modeling long-term disease trajectories. By grounding reinforcement learning in human expertise through a supervised screening step, the authors successfully navigated high-dimensional clinical action spaces.
Despite these promising results, limitations remain. The sample size of 6,000 patients, while large for clinical research, is modest for deep learning, leading to overlapping confidence intervals. Furthermore, the current model assumes the Markov property; future iterations could incorporate Recurrent Neural Networks (RNNs) to better account for a patient's full medical history. Finally, while retrospective evaluation is a vital first step, prospective clinical trials are necessary to validate these data-driven recommendations in real-world settings.
Summary updated