Abstract
Dynamic treatment regimes have been proposed to personalize treatment decisions by utilizing historical patient data, but they may not always improve on the current standard of care. It is thus meaningful to integrate the standard of care into the evaluation of treatment strategies, and previous works have suggested doing so through the concept of clinical utility. Here we will focus on the comparative component of clinical utility as the average outcome had the full population received treatment based on the proposed dynamic treatment regime in comparison to the full population receiving the "standard" treatment assignment mechanism, such as a physician's choice. Clinical trials to evaluate clinical utility are rarely conducted, and thus, previous works have proposed an emulated clinical trial framework using observational data. However, only one simple estimator was previously suggested, and the practical details of how one would conduct this emulated trial were not detailed. Here, we illuminate these details and propose several estimators of clinical utility based on estimators proposed in the dynamic treatment regime literature. We illustrate the considerations and the estimators in a real data example investigating treatment rules for rheumatoid arthritis, where we highlight that in addition to the standard of care, the current medical guidelines should also be compared to any estimated "optimal" decision rule.
Introduction
In the field of personalized medicine, researchers often develop Dynamic Treatment Regimes (DTRs)—sets of decision rules that guide clinicians on treatment choices based on a patient's specific history and biomarkers. While the primary goal of DTR research is typically to find an "optimal" rule that maximizes health outcomes, a significant practical hurdle remains: an optimal rule is only clinically valuable if it outperforms the current Standard of Care (SOC).
This paper addresses this gap by refining the concept of clinical utility—the incremental benefit of a specific DTR compared to the existing physician-led assignment mechanism. By utilizing an emulated clinical trial framework, the authors provide a practical roadmap for evaluating new treatment rules against meaningful real-world benchmarks using observational data.
Methodology
The paper employs Nonparametric Structural Equation Models (NPSEMs) and Directed Acyclic Graphs (DAGs) to identify causal effects from observational datasets. The causal structure assumes that baseline confounders ($Z$) influence both the treatment assignment ($T$) and the final health outcome ($Y$).
To estimate clinical utility, the authors propose and compare four primary estimators:
- Inverse Probability Weighting (IPW): Includes a non-binary version (
ipwnb) and a simplified binary version (ipwb) that models the probability of a patient's observed treatment matching the DTR's recommendation. - G-computation: Includes non-binary (
gcnb) and binary (gcb) versions that use outcome models to predict results under different treatment scenarios.
For identifying the "optimal" DTR ($f_{opt}$), the study utilizes Random Fourier Features (RFF), a machine learning technique that enables linear models to capture complex, non-linear relationships efficiently.
Simulation Results
The estimators were tested across various synthetic scenarios and sample sizes ($n=200$ to $n=2000$). The results highlight critical trade-offs in estimator stability and bias.
As shown in the simulation table, all four estimators demonstrate minimal bias and approximately 95% confidence interval coverage at large sample sizes ($n=2000$). However, at smaller sample sizes ($n=200$), the performance of IPW estimators degrades significantly, showing higher bias and lower coverage. In contrast, G-computation remained more reliable in smaller cohorts, provided the underlying outcome model was correctly specified.
Case Study: Rheumatoid Arthritis
The framework was applied to a real-world dataset of 1,966 patients from the Danish National Laboratory Database, focusing on treatment switches for patients failing first-line therapy. The primary outcome was the change in C-reactive protein (CRP), a marker of inflammation.
Key insights from the case study include:
- The "Optimal" Isn't Always Better: The data-driven "optimal" rule ($f_{opt}$) failed to significantly outperform the current SOC. This suggests that physicians may be utilizing unmeasured clinical intuition that simple data models cannot yet replicate.
- Guidelines as DTRs: When treating the EULAR 2023 medical guidelines as a DTR ($f_{cgl}$), the researchers found a significant improvement over the historical SOC. Furthermore, these expert-derived guidelines actually outperformed the machine-learned "optimal" rule in this context.
- The Importance of "Time Zero": The authors emphasize that defining when a patient becomes eligible for a treatment switch is critical. Emulating DTR trials requires ensuring all necessary biomarkers were measured immediately prior to the decision point.
Conclusion
This research highlights that even "optimal" data-driven rules must be rigorously benchmarked against the standard of care and established medical guidelines. While G-computation offers a robust estimation tool for smaller datasets, the success of any emulated trial depends heavily on accounting for the clinical nuances captured by experienced physicians. Future work will focus on extending this framework to multi-stage treatment decisions and ensuring the transportability of utility estimates across different healthcare systems.
Summary updated