Standard machine learning models and clinical guidelines often treat medical decisions as isolated, single-shot events. As noted by Siqi Liu et al. in their comprehensive review of critical care systems, this "static treatment assumption" fails to account for the dynamic reality of medicine: a patient's response to an initial drug dosage changes their physiological state, which in turn dictates what the next dosage should be.
To bridge this gap, researchers are developing frameworks for Dynamic Treatment Regimes (DTRs)—sequential decision rules that adapt medical interventions over time based on a patient's evolving health status.
The Challenge of Time-Varying Confounding
Why can we not simply apply standard, single-stage causal inference to longitudinal medical records? The core hurdle is a complex feedback loop known as time-varying confounding.
In a sequential setting, a patient's history determines both the current treatment assignment and future clinical outcomes. For example, in mechanical ventilation weaning, sedation dosages have cumulative, delayed physiological impacts on a patient's readiness to breathe independently (A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Units). If a model treats each decision point as independent, it will fail to capture how a treatment administered today alters the patient's state tomorrow, thereby confounding all subsequent treatment decisions. As Zhaozhi Qian et al. point out, traditional point-treatment frameworks are fundamentally blind to these compounding long-term effects.
Formalizing the Sequential Decision Process
To capture this feedback loop, researchers map the clinical environment to sequential decision-making frameworks, most notably Reinforcement Learning (RL) and Markov Decision Processes (MDPs).
In this mapping:
- States represent the patient's evolving clinical profile (e.g., demographics, lab results, vitals).
- Actions represent medical interventions (e.g., titrating a drug, switching therapies, or weaning a ventilator).
- Rewards represent long-term patient outcomes (e.g., survival, reduction in symptom severity).
By framing the problem this way, the goal shifts from predicting an immediate clinical outcome to finding a policy—a sequence of "if-then" rules—that maximizes the cumulative reward over a patient's entire care trajectory. For instance, Ning Liu et al. demonstrate that Deep Q-Networks (DQNs) can ingest retrospective registry data to estimate long-term value functions, discovering multi-stage drug combinations that optimize survival for bone marrow transplant patients.
[Patient State S_t] ---> [Policy / DTR] ---> [Action A_t (e.g., Dosage)]
^ |
| v
[Updated State S_{t+1}] <--- [Patient Physiology] <----+
This sequential framing allows models to discover non-obvious, highly effective treatment paths. Jie Yang et al. highlight how RL agents can learn optimal titration strategies for fluids and vasopressors in sepsis patients by analyzing historical trajectories. Strikingly, multiple studies show that when human clinical decisions deviate from these AI-optimized paths, patient mortality rates increase (Jie Yang et al.; Matthieu Komorowski et al.). This suggests that sequential algorithms can uncover subtle physiological patterns and long-term compounding benefits that elude human intuition.
Designing Trials for Dynamic Strategies
Discovering these regimes requires specialized data collection strategies. Traditional randomized controlled trials (RCTs) are designed to evaluate the final efficacy of a fixed, completed intervention protocol rather than optimizing its individual components or timing (Niranjan Bidargaddi et al.).
To address this, researchers utilize Sequential Multiple Assignment Randomized Trials (SMARTs) and Micro-Randomized Trials (MRTs). In a SMART, patients move through multiple stages of treatment, and at each stage, they are re-randomized based on their response to the previous stage. This directly generates the data needed to construct optimal adaptive pathways.
However, a scoping review by Nikki L. B. Freeman et al. revealed a significant "translation gap" in the literature: nearly half of published SMART analyses fail to evaluate the integrated treatment sequences they were designed to test, instead treating the study as a series of disconnected single-stage trials. Closing this gap requires a tighter integration of machine learning and sequential causal inference to fully exploit adaptive trial data.
Advanced Architectures for Clinical Realities
As the field matures, researchers are moving beyond basic MDPs to handle the messy realities of clinical data, such as partial observability, high dimensionality, and variable sequence lengths:
- Partial Observability: Clinical data is often noisy and incomplete. To manage this, Luchen Li et al. model sequential treatments as Partially Observable Markov Decision Processes (POMDPs). Their framework combines variational generative models with parallel look-ahead search trees to optimize sepsis dosages under deep uncertainty.
- High-Dimensional Inference: When patient characteristics outnumber the sample size, traditional statistical inference breaks down. Muxuan Liang introduces a "split-and-pooled de-correlated score" method to reliably estimate and perform hypothesis testing on high-dimensional individualized treatment rules from observational data.
- Autoregressive Sequence Modeling: Rather than relying on explicit causal graphs, Daniel Jiwoong Im et al. demonstrate that Transformer-based autoregressive models can act as causal inference engines. By "sequencifying" causal graphs into token streams, they can naturally handle variable-length action sequences where the number of treatment steps depends entirely on prior patient responses.
Key Foundational Work
Identifying Optimal Sequential Decisions (2008) — contributed 4 concept elements to this direction.