TL;DR
This paper outlines a pilot clinical trial protocol for DEMONSTRATE, an ML-based clinical decision support (CDS) tool designed to predict patients' risk of opioid overdose within three months. The trial aims to evaluate the tool's usability, acceptability, feasibility, and effectiveness in real-world primary care settings, with the ultimate goal of reducing opioid-related harm.
What Problem Does This Paper Address?
The opioid epidemic remains a critical public health crisis, with drug overdoses, largely driven by opioids, claiming over half a million lives in the US annually. While existing prevention strategies like prescription drug monitoring programs (PDMPs) and adherence to prescribing guidelines have been implemented, they often fall short in accurately identifying patients at the highest risk of overdose or opioid use disorder (OUD). These traditional methods have shown limited impact on mortality and can fail to capture a significant portion of at-risk individuals. Current surveillance systems are often retrospective and siloed, identifying risk only after adverse events occur. This paper addresses the urgent need for proactive, data-driven approaches that can identify high-risk patients before harm occurs, integrating seamlessly into clinical workflows to enable timely interventions.
Methods
The DEMONSTRATE trial employs a single-arm, pre-post implementation design with a mixed-methods evaluation. The core of the intervention is the "Overdose Prevention Alert," a machine learning (ML)-driven clinical decision support (CDS) tool embedded within the electronic health record (EHR). This tool triggers when a primary care provider (PCP) signs an opioid prescription for a patient identified by the ML algorithm as high-risk for overdose. The ML model itself was developed and validated using existing EHR data, incorporating 57 predictors across patient demographics, healthcare utilization, prescription characteristics, and comorbid diagnoses.
The study utilizes a user-centered design approach for the alert's interface and backend, involving PCPs and human factors experts to ensure workflow integration and minimize unintended consequences. The alert has two versions: one recommending naloxone ordering if no recent order is found, and another prompting confirmation of existing naloxone if an order is present. The trial will assess effectiveness by comparing patient-level outcomes before and after implementation, focusing on a composite measure of six favorable indicators (e.g., naloxone access, absence of overdose diagnoses/hospitalizations). PCP-level outcomes will be evaluated through questionnaires and interviews to gauge usability, acceptability, and feasibility.
Datasets
The ML model underlying the Overdose Prevention Alert was developed and validated using 2016-2020 University of Florida (UF) Health EHR data. This training cohort included over 130,000 adult patients (aged ≥18) who received at least one outpatient opioid prescription. The data included patient demographics, healthcare utilization, prescription data, and comorbid diagnoses. The current pilot trial is being conducted within 13 UF Health Gainesville primary care clinics.
Key Insights and Ideas
- Proactive Risk Identification: The study moves beyond retrospective risk assessment to a proactive, ML-driven approach that aims to identify high-risk patients before an overdose event occurs.
- EHR Integration is Crucial: Embedding the ML model into a CDS tool within the existing EHR workflow is key to enabling real-time decision support and facilitating adoption by PCPs.
- User-Centered Design for Adoption: The development process emphasizes human factors and PCP input to create an alert that is usable, understandable, and minimizes alert fatigue, addressing common barriers to CDS tool implementation.
- Focus on Actionable Interventions: The alert not only identifies risk but also provides actionable recommendations, such as ordering naloxone or discussing risks with the patient, aiming to drive concrete harm-reduction behaviors.
- Mixed-Methods for Comprehensive Evaluation: Combining quantitative outcome measures with qualitative assessments of PCP experience provides a holistic understanding of the tool's impact and feasibility.
Key Results
As this is a protocol for a pilot trial, definitive results are not yet available. However, the expected outcomes are:
- Usability, Acceptability, and Feasibility: The study anticipates providing insights into how well the Overdose Prevention Alert can be integrated into routine primary care workflows and whether PCPs find it acceptable and feasible to use.
- Preliminary Effectiveness: The trial aims to generate preliminary effect-size estimates for the composite patient-level outcome measure, indicating the potential impact of the ML-driven CDS tool on reducing opioid-related harm.
- Alert Penetration and Usage Patterns: Quantitative metrics will detail how often the alert fires, how PCPs interact with it (e.g., override reasons, accepted recommendations), and the overall penetration of the tool within the participating clinics.
- Insights for Future Trials: The pilot study is designed to inform the design of a larger, fully powered cluster-randomized trial by providing essential data on effect sizes, clustering, and operational feasibility.
Limitations and Future Work
- Single-Arm Design: The study's single-arm, pre-post design means that observed changes in outcomes cannot be definitively attributed to the intervention alone, as secular trends or other concurrent events could also play a role.
- Legacy Alert Co-existence: The continued operation of a legacy rule-based naloxone alert alongside the new ML alert introduces analytic complexity and may influence PCP behavior.
- Generalizability: The ML model was developed using data exclusively from UF Health, which may limit the generalizability of findings to other healthcare systems.
- Potential Underpowering: The pilot study may be underpowered to detect statistically significant differences in individual patient-level outcomes, especially given potential declines in overall opioid prescribing.
- Future Work: The findings from this pilot trial are intended to directly inform the design of a future cluster-randomized or pragmatic trial. This future work will aim to provide more robust evidence on the effectiveness of the ML-driven CDS tool and refine its implementation strategy for broader deployment. The study also aims to disseminate findings to various stakeholders to guide future CDS implementation strategies.
Summary updated