Abstract
Dynamic treatment regimes are sequential decision rules that adapt treatment according to individual time-varying characteristics and outcomes to achieve optimal effects, with applications in precision medicine, personalized recommendations, and dynamic marketing. Estimating optimal dynamic treatment regimes via sequential randomized trials might face costly and ethical hurdles, often necessitating the use of historical observational data. In this work, we utilize proximal causal inference framework for learning optimal dynamic treatment regimes when the unconfoundedness assumption fails. Our contributions are four-fold: (i) we propose three nonparametric identification methods for optimal dynamic treatment regimes; (ii) we establish the semiparametric efficiency bound for the value function of a given regime; (iii) we propose a (K+1)-robust method for learning optimal dynamic treatment regimes, where K is the number of stages; (iv) as a by-product for marginal structural models, we establish identification and estimation of counterfactual means under a static regime. Numerical experiments validate the efficiency and multiple robustness of our proposed methods.
Introduction
In fields such as precision medicine, personalized marketing, and adaptive education, decision-making is rarely a one-off event. Instead, it involves a sequence of interventions tailored to the evolving state of the subject. These sequences are known as Dynamic Treatment Regimes (DTRs). For instance, a clinician may adjust a patient's medication dosage based on their response to a previous treatment stage.
While randomized controlled trials (RCTs) are the gold standard for identifying optimal DTRs, they are often prohibitively expensive or ethically unfeasible. Consequently, researchers must rely on observational data. The primary challenge in this context is unmeasured confounding: the presence of hidden factors that influence both the treatment assignment and the outcome. If these confounders are ignored, the resulting policy will be biased and potentially harmful. This paper introduces a robust framework to address these challenges using proximal causal inference.
The Challenge of Unmeasured Confounding
Traditional causal inference methods rely on the "unconfoundedness" assumption, which posits that all variables influencing both treatment and outcome have been measured. In complex, real-world longitudinal studies, this assumption is frequently violated.
Proximal Causal Inference offers a solution by utilizing "proxy" variables—observed variables that serve as stand-ins for unmeasured confounders. For example, while a patient's exact physiological state might be unmeasured, blood pressure and heart rate could serve as proxies. However, extending proximal inference to multi-stage DTRs is difficult due to "error amplification," where minor mathematical inaccuracies in early stages compound exponentially as the decision chain progresses.
Methodology: PHA and PMR
The authors, Gao, Bai, and Cui, propose three primary innovations to stabilize and optimize sequential decision-making under confounding:
1. Proximal Hybrid Augmentation (PHA)
Standard proximal methods typically focus on either modeling the outcome (outcome bridge functions) or the treatment assignment (weight bridge functions). The PHA method integrates both, distributing the mathematical burden across two different types of bridge functions. This hybrid approach significantly reduces the instability and noise typically associated with multi-stage proximal models.
2. The (K+1)-Robust Estimator (PMR)
The centerpiece of the paper is the Proximal Multiply Robust (PMR) estimator. In a decision process with $K$ stages, the authors identify $K+1$ distinct ways to specify the underlying models. The PMR estimator is designed to be consistent if any one of these $K+1$ sub-models is correctly specified. This provides a significant "safety net" for researchers; even if several parts of the longitudinal model are misspecified, the resulting policy remains reliable.
3. Semiparametric Efficiency
The authors establish the semiparametric efficiency bound for the value function of a regime. This proof ensures that the PMR estimator is not just robust, but also statistically optimal—extracting the maximum possible information from the available data with the lowest possible variance.
Experimental Validation
The framework was tested using a synthetic Data Generating Process (DGP) simulating a two-stage decision process with 11 binary variables. The researchers intentionally introduced model misspecifications to test the limits of the PMR estimator.
Key findings from the numerical experiments include:
- Superior Robustness: Even when treatment models were incorrectly specified, the PMR estimator maintained near-zero "regret" (the gap between the learned policy and the true optimal policy).
- High Accuracy: The PMR method's error rates were 100x to 1000x lower than "Naive" approaches that ignored hidden confounders.
- Efficiency Gains: Even under perfect model specification, the PMR method demonstrated lower variance compared to simpler proximal alternatives, confirming its theoretical efficiency.
Discussion and Limitations
While the PMR framework represents a major advancement, it faces practical hurdles. The implementation requires solving complex Fredholm integral equations, which are computationally demanding. Furthermore, the validity of the entire system hinges on the quality of the proxies; if the chosen proxies do not sufficiently relate to the hidden confounders, the identification of the optimal regime fails.
Future research directions include integrating machine learning to automate the discovery of bridge functions and extending the framework to survival analysis, where outcomes are measured as time-to-event data.
Conclusion
This work provides a rigorous mathematical foundation for learning optimal sequential policies in the presence of hidden bias. By introducing (K+1)-robustness and the PHA method, the authors offer a practical and theoretically sound path forward for precision medicine and other domains where sequential, data-driven decision-making is critical.
Summary updated