Abstract
Machine learning-based clinical decision support tools for sepsis create opportunities to identify at-risk patients and initiate treatments earlier, critical to improving sepsis outcomes. Increasing use of such systems necessitates quantifying and understanding provider adoption. Using real-time provider interactions with a sepsis early detection tool (Targeted Real-time Early Warning System) deployed at five hospitals over a two-year period (469,419 screened encounters, 9,805 (2.1%) retrospectively-identified sepsis cases), we found high sensitivity (82% of sepsis cases identified), high adoption rates (89% of alerts evaluated by a physician or advanced practice provider and 38% of evaluated alerts confirmed) and an association between use of the tool and earlier treatment of sepsis patients (1.85 (95% CI:1.66-2.00) hour reduction in median time to first antibiotics order). Further, we found that provider-related factors were strongly associated with adoption. Beyond improving system performance, efforts to improve adoption should focus on provider knowledge, experience, and perceptions of the system.
Introduction
Sepsis is a critical medical emergency where the body’s response to infection triggers widespread organ failure. Because every hour of delay in antibiotic treatment significantly increases mortality, early detection is paramount. While machine learning (ML) models excel at identifying subtle patterns in Electronic Health Record (EHR) data that signal early sepsis, they often fail in the "last mile" of clinical implementation due to "alert fatigue"—a phenomenon where clinicians ignore frequent, non-contextual notifications.
This study evaluates the Targeted Real-time Early Warning System (TREWS), a system designed to overcome these barriers. By analyzing its deployment across five hospitals over two years, the researchers investigated whether clinicians actually use the tool, if it leads to faster treatment, and what specific factors drive clinician trust.
Methodology and System Design
TREWS was deployed across two academic and three community hospitals within the Johns Hopkins Health System. The system continuously scans EHR data—including vitals, laboratory results, and clinical notes—to generate patient-specific risk scores.
Unlike traditional interruptive systems that rely on pop-up windows, TREWS utilizes a passive workflow. It visually flags high-risk patients within the EHR, allowing providers to click into a dedicated interface that provides transparency by showing the specific clinical factors that triggered the alert.
To evaluate the system, the authors analyzed 469,419 screened patient encounters (including 9,805 retrospectively identified sepsis cases). They utilized quantile regression to estimate the reduction in time to antibiotic administration and logistic regression to identify factors associated with alert adoption.
Key Results and Clinical Impact
The study found that TREWS achieved high performance and significant clinical engagement:
- High Sensitivity: The system identified 82% of sepsis cases.
- Exceptional Adoption: 89% of all alerts were evaluated by a provider, with 73% evaluated within the critical three-hour window. This is remarkably high compared to other clinical decision support tools, which often see adoption rates between 6% and 45%.
- Faster Treatment: When a provider confirmed a TREWS alert within three hours, there was a 1.85-hour reduction in the median time to the first antibiotic order compared to cases where alerts were ignored or dismissed.
- Precision: 38% of alerts evaluated by a physician were confirmed as sepsis, indicating a high "hit rate" that helps mitigate alert fatigue.
Drivers of Adoption: The Human Factor
A central finding of the research is that provider-related factors were stronger predictors of adoption than the actual severity of the patient's symptoms.
- Situational and Learned Trust: Providers in the Emergency Department (ED) were significantly more likely to engage with alerts, likely due to the high-uncertainty environment where protocolized alerts are more welcome. Furthermore, providers who had used the tool previously were more likely to use it again, suggesting that trust is "learned" as clinicians become familiar with the system's accuracy.
- The Atypical Case Challenge: Alerts were more likely to be dismissed in younger patients or those lacking "classic" sepsis symptoms (like high fever). This suggests that clinicians may currently use ML tools to confirm existing suspicions rather than to catch the "atypical" cases where the algorithm actually adds the most value.
- Workflow Integration: The authors credit the high adoption rate to the tool’s passive nature and the provision of "alert context," which avoids the frustration of "black box" notifications.
Conclusion and Future Directions
The success of TREWS demonstrates that effective clinical ML deployment requires more than just algorithmic accuracy; it requires seamless workflow integration and transparency. While the study is limited by its retrospective labeling and focus on a single health system, it highlights a shift in how we should approach clinical AI. Future efforts should focus less on tweaking algorithms and more on provider education—specifically teaching clinicians how to interpret alerts for patients who do not present with typical symptoms. Ensuring that ML tools are viewed as collaborative partners rather than interruptive nuisances is key to improving patient outcomes.
Summary updated