Abstract
Stabilized dynamic treatment regimes are sequential decision rules for individual patients that not only adaptive throughout the disease progression but also remain consistent over time in format. The estimation of stabilized dynamic treatment regimes becomes more complicated when the clinical outcome of interest is a survival time subject to censoring. To address this challenge, we propose two novel methods, censored shared-Q-learning and censored shared-O-learning. Both methods incorporate clinical preferences into a qualitative rule, where the parameters indexing the decision rules are shared across different stages and estimated simultaneously. We use extensive simulation studies to demonstrate the superior performance of the proposed methods. The methods are further applied to the Framingham Study to derive treatment rules for cardiovascular disease.
Introduction
In the management of chronic conditions, clinicians must make a series of decisions over time based on a patient's evolving health status. A Dynamic Treatment Regime (DTR) provides a formal framework for these decisions. However, standard DTRs often result in different decision rules for every stage of a disease—for example, recommending different blood pressure thresholds in year one versus year five. This lack of consistency can hinder clinical adoption.
Stabilized Dynamic Treatment Regimes (SDTRs) address this by enforcing a shared rule format across all time points. While SDTRs improve interpretability, their estimation is significantly complicated when the primary outcome is survival time subject to censoring—a common occurrence in clinical trials where the exact event time is unknown for patients who exit the study early. This paper introduces two frameworks, censored shared-Q-learning (CSQL) and censored shared-O-learning (CSOL), to bridge the gap between complex personalized medicine and practical implementation in survival analysis.
Problem Formulation
The primary objective is to derive a sequence of decision rules that maximize the expected survival time (the "value function") while maintaining a consistent rule structure across $K$ stages. The challenge lies in the fact that survival data is often right-censored. To handle this, the proposed methods utilize Inverse Probability of Censoring Weights (IPCW), which re-weights observed event times to account for the distribution of censored individuals, ensuring that the estimated treatment effects remain unbiased.
Proposed Methodology
Censored Shared-Q-learning (CSQL)
CSQL is an indirect, regression-based approach. It involves modeling "Q-functions," which represent the expected future reward (survival time) given a specific state and action.
- Mechanism: The method uses an iterative algorithm to estimate parameters that are shared across all stages.
- Censoring Handling: By incorporating IPCW, CSQL adjusts the regression targets to account for the probability of a patient remaining uncensored.
- Strengths: CSQL is highly efficient when the underlying regression models for the disease progression are correctly specified.
Censored Shared-O-learning (CSOL)
CSOL is a direct, outcome-weighted learning approach that bypasses the need to model intermediate rewards.
- Mechanism: It directly maximizes the population's expected survival time. Because the objective function in O-learning is typically non-differentiable (due to the indicator functions in the value estimator), the authors employ a soft-minimum function and a concave surrogate loss (similar to logistic loss).
- Optimization: This "soft-minimum trick" allows the use of standard optimization algorithms like BFGS to find a single rule that performs well across multiple stages.
- Strengths: CSOL is notably more robust than CSQL when the researcher's model of the disease is misspecified.
Experimental Evaluation
Simulation Studies
The methods were tested using a synthetic diabetes simulation mimicking 10 decision points, tracking variables such as A1c levels and blood pressure.
- Scenario 1 (Correct Specification): CSQL performed optimally when the model assumptions matched the data generation.
- Scenario 2 (Misspecification): When the underlying model was incorrect, CSQL's performance dropped significantly (mean survival of 5.73). In contrast, CSOL achieved a mean survival of 14.67, demonstrating its robustness and superior ability to find effective rules under uncertainty.
Application: The Framingham Heart Study
The frameworks were applied to data from 2,236 subjects in the Framingham Heart Study across five biannual visits to determine treatment rules for cardiovascular disease (CVD).
- Clinical Relevance: Both CSQL and CSOL derived rules that significantly outperformed standard care. Patients whose actual treatments aligned with the CSOL recommendations showed significantly better survival outcomes ($p < 0.001$).
- Efficiency: Interestingly, CSOL recommended medication less frequently than current clinical practice while achieving similar or better survival benefits, suggesting a more targeted and efficient use of medical interventions.
Discussion and Conclusion
This research demonstrates that stabilized rules—while potentially sacrificing a small degree of theoretical optimality compared to stage-specific rules—offer a much higher degree of clinical utility. The introduction of CSOL provides a robust alternative to regression-based methods, particularly in complex real-world scenarios where the true disease model is unknown.
Future work may expand these linear decision rules into tree-based models to further enhance interpretability. Additionally, the integration of Augmented IPCW (AIPCW) could improve statistical efficiency by allowing the models to learn from the entire patient cohort, including those whose treatments did not match the proposed regime.
Summary updated