Protocol for a Single-Arm Pilot Clinical Trial: Developing and Evaluating a Machine Learning Opioid Prediction & Risk-Stratification E-Platform (DEMONSTRATE)

Jung Hwa Hong, Debbie L. Wilson, Khoa A. Nguyen, Walid F. Gellad, Julie Diiulio, Laura G. Militello, Shunhua Yan, Christopher A. Harle, Danielle Nelson, Eric I. Rosenberg, Siegfried Schmidt, Chung‐Chou H. Chang, Gerald Cochran, Yonghui Wu, Stephanie A. S. Staras, Courtney C. Kuza, Wei‐Hsuan Lo‐CiganicJournal of Clinical Medicine 2025 · DOI

TL;DR

This paper outlines a pilot clinical trial protocol for DEMONSTRATE, an ML-based clinical decision support (CDS) tool designed to predict patients' risk of opioid overdose within three months. The trial aims to evaluate the tool's usability, acceptability, feasibility, and effectiveness in real-world primary care settings, with the ultimate goal of reducing opioid-related harm.

What Problem Does This Paper Address?

The opioid epidemic remains a critical public health crisis, with drug overdoses, largely driven by opioids, claiming over half a million lives in the US annually. While existing prevention strategies like prescription drug monitoring programs (PDMPs) and adherence to prescribing guidelines have been implemented, they often fall short in accurately identifying patients at the highest risk of overdose or opioid use disorder (OUD). These traditional methods have shown limited impact on mortality and can fail to capture a significant portion of at-risk individuals. Current surveillance systems are often retrospective and siloed, identifying risk only after adverse events occur. This paper addresses the urgent need for proactive, data-driven approaches that can identify high-risk patients before harm occurs, integrating seamlessly into clinical workflows to enable timely interventions.

Methods

The DEMONSTRATE trial employs a single-arm, pre-post implementation design with a mixed-methods evaluation. The core of the intervention is the "Overdose Prevention Alert," a machine learning (ML)-driven clinical decision support (CDS) tool embedded within the electronic health record (EHR). This tool triggers when a primary care provider (PCP) signs an opioid prescription for a patient identified by the ML algorithm as high-risk for overdose. The ML model itself was developed and validated using existing EHR data, incorporating 57 predictors across patient demographics, healthcare utilization, prescription characteristics, and comorbid diagnoses.

The study utilizes a user-centered design approach for the alert's interface and backend, involving PCPs and human factors experts to ensure workflow integration and minimize unintended consequences. The alert has two versions: one recommending naloxone ordering if no recent order is found, and another prompting confirmation of existing naloxone if an order is present. The trial will assess effectiveness by comparing patient-level outcomes before and after implementation, focusing on a composite measure of six favorable indicators (e.g., naloxone access, absence of overdose diagnoses/hospitalizations). PCP-level outcomes will be evaluated through questionnaires and interviews to gauge usability, acceptability, and feasibility.

Datasets

The ML model underlying the Overdose Prevention Alert was developed and validated using 2016-2020 University of Florida (UF) Health EHR data. This training cohort included over 130,000 adult patients (aged ≥18) who received at least one outpatient opioid prescription. The data included patient demographics, healthcare utilization, prescription data, and comorbid diagnoses. The current pilot trial is being conducted within 13 UF Health Gainesville primary care clinics.

Key Insights and Ideas

  • Proactive Risk Identification: The study moves beyond retrospective risk assessment to a proactive, ML-driven approach that aims to identify high-risk patients before an overdose event occurs.
  • EHR Integration is Crucial: Embedding the ML model into a CDS tool within the existing EHR workflow is key to enabling real-time decision support and facilitating adoption by PCPs.
  • User-Centered Design for Adoption: The development process emphasizes human factors and PCP input to create an alert that is usable, understandable, and minimizes alert fatigue, addressing common barriers to CDS tool implementation.
  • Focus on Actionable Interventions: The alert not only identifies risk but also provides actionable recommendations, such as ordering naloxone or discussing risks with the patient, aiming to drive concrete harm-reduction behaviors.
  • Mixed-Methods for Comprehensive Evaluation: Combining quantitative outcome measures with qualitative assessments of PCP experience provides a holistic understanding of the tool's impact and feasibility.

Key Results

As this is a protocol for a pilot trial, definitive results are not yet available. However, the expected outcomes are:

  • Usability, Acceptability, and Feasibility: The study anticipates providing insights into how well the Overdose Prevention Alert can be integrated into routine primary care workflows and whether PCPs find it acceptable and feasible to use.
  • Preliminary Effectiveness: The trial aims to generate preliminary effect-size estimates for the composite patient-level outcome measure, indicating the potential impact of the ML-driven CDS tool on reducing opioid-related harm.
  • Alert Penetration and Usage Patterns: Quantitative metrics will detail how often the alert fires, how PCPs interact with it (e.g., override reasons, accepted recommendations), and the overall penetration of the tool within the participating clinics.
  • Insights for Future Trials: The pilot study is designed to inform the design of a larger, fully powered cluster-randomized trial by providing essential data on effect sizes, clustering, and operational feasibility.

Limitations and Future Work

  • Single-Arm Design: The study's single-arm, pre-post design means that observed changes in outcomes cannot be definitively attributed to the intervention alone, as secular trends or other concurrent events could also play a role.
  • Legacy Alert Co-existence: The continued operation of a legacy rule-based naloxone alert alongside the new ML alert introduces analytic complexity and may influence PCP behavior.
  • Generalizability: The ML model was developed using data exclusively from UF Health, which may limit the generalizability of findings to other healthcare systems.
  • Potential Underpowering: The pilot study may be underpowered to detect statistically significant differences in individual patient-level outcomes, especially given potential declines in overall opioid prescribing.
  • Future Work: The findings from this pilot trial are intended to directly inform the design of a future cluster-randomized or pragmatic trial. This future work will aim to provide more robust evidence on the effectiveness of the ML-driven CDS tool and refine its implementation strategy for broader deployment. The study also aims to disseminate findings to various stakeholders to guide future CDS implementation strategies.

Summary updated

Abstract

<b>Background/Objectives</b>: The Developing and Evaluating a Machine Learning Opioid Prediction & Risk-Stratification E-Platform (DEMONSTRATE) trial aims to assess the usability, acceptability, feasibility, and effectiveness of implementing a machine learning (ML)-based clinical decision support (CDS) tool-the Overdose Prevention Alert-which predicts a patient's risk of opioid overdose within three months. <b>Methods</b>: This single-arm study uses a pre-post implementation design with mixed-methods evaluation in 13 University of Florida Health, Gainesville, internal medicine and family medicine clinics. Eligible patients are aged ≥18 years, received an opioid prescription within the year prior to their upcoming primary care visit, are not receiving hospice care, do not have a malignant cancer diagnosis, and are identified by the ML algorithm as high risk for overdose. The Overdose Prevention Alert triggers when a primary care provider (PCP) signs an opioid order in electronic health records. We will evaluate effectiveness by comparing pre- and post-implementation outcomes using a composite patient-level measure defined by the presence of any of the following 6 favorable indicators: (1) evidence of naloxone access; (2) absence of opioid overdose diagnoses and naloxone administration; (3) absence of emergency department (ED) visits or hospitalizations due to opioid overdose or opioid use disorder (OUD); (4) absence of overlapping opioid and benzodiazepine use within a 7-day window; (5) absence of opioid use ≥50 morphine milligram equivalent daily average; (6) receipt of referrals to non-pharmacological pain management. Additional quantitative metrics will include alert penetration, usage patterns, and clinical actions taken. Usability and acceptability will be assessed using a 12-item questionnaire for PCPs and semi-structured interviews. <b>Expected Results</b>: The trial will provide insights into real-world ML-driven CDS implementation and inform future strategies to reduce opioid-related harm.
BibTeX
@article{hong2025protocol,
  author = {Jung Hwa Hong and Debbie L. Wilson and Khoa A. Nguyen and Walid F. Gellad and Julie Diiulio and Laura G. Militello and Shunhua Yan and Christopher A. Harle and Danielle Nelson and Eric I. Rosenberg and Siegfried Schmidt and Chung‐Chou H. Chang and Gerald Cochran and Yonghui Wu and Stephanie A. S. Staras and Courtney C. Kuza and Wei‐Hsuan Lo‐Ciganic},
  title = {{Protocol for a Single-Arm Pilot Clinical Trial: Developing and Evaluating a Machine Learning Opioid Prediction \&amp; Risk-Stratification E-Platform (DEMONSTRATE)}},
  journal = {Journal of Clinical Medicine},
  year = {2025},
  doi = {10.3390/jcm14238522},
}
Figure