Effect of Digital Early Warning Scores on Hospital Vital Sign Observation Protocol Adherence: Stepped-Wedge Evaluation

David Wong, Timothy Bonnici, Stephen Gerry, Jacqueline Birks, Peter WatkinsonJournal of Medical Internet Research 2024 · DOI

TL;DR

This study investigated whether implementing a digital Early Warning Score (EWS) system in hospitals improved patient monitoring and outcomes compared to traditional paper charts. Despite strong user engagement and good system usability, the digital system did not lead to significant improvements in patient outcomes like time to next observation, hospital mortality, or length of stay. This suggests that simply digitizing EWS might not be enough to improve care, and future efforts should focus on integrating these systems with new clinical pathways to drive behavioral change.

What Problem Does This Paper Address?

Unrecognized clinical deterioration in hospitalized patients is a significant problem, leading to avoidable mortality. Early Warning Scores (EWS) were developed to help identify patients at risk by scoring deviations in vital signs, triggering escalation protocols for timely intervention. While EWS are widely adopted, their clinical effectiveness remains mixed. Challenges include errors in EWS calculation, delays in documentation, and poor adherence to escalation protocols. Digital EWS systems aim to automate calculations and provide timely information, but robust evidence on their impact on clinical outcomes and patient management is lacking. This paper addresses the question of whether a digital EWS system, specifically the SEND system, leads to better recognition of deterioration and closer adherence to escalation protocols, ultimately improving patient care.

Methods

The researchers employed a stepped-wedge cluster randomized trial design. This means that instead of all hospitals switching to the digital system at once, the implementation was staggered across different wards and hospitals over time. The study involved four hospitals within a single UK National Health Service trust.

  • Control Arm: Vital signs were recorded using traditional paper observation charts.
  • Intervention Arm: A digital EWS system called SEND (System for Electronic Notification and Documentation) was used. This system automatically calculates EWS based on manually entered vital sign data and displays relevant escalation protocol advice.

The study design ensured that at any given time, some wards were in the control arm while others were in the intervention arm. This allowed for a direct comparison of outcomes between the two approaches. The primary outcome measured was the time to next observation (TTNO) after a patient's vital signs indicated a high risk (EWS ≥ 3). Secondary outcomes included time to death, length of hospital stay, and time to unplanned intensive care unit (ICU) admission. Statistical analysis used a mixed-effects Cox model to account for the clustered nature of the data and competing events. System usability was assessed using the System Usability Scale (SUS).

Datasets

The study utilized data from 90,262 admissions to study wards within the Oxford University Hospitals Foundation NHS Trust between January 2015 and September 2016.

  • Control Arm Data: Collected from paper charts and transcribed by research assistants.
  • Intervention Arm Data: Collected directly from the SEND digital system.

The primary analysis included 12,802 admissions, specifically 1084 from the control arm and 11,718 from the intervention arm, where a triggering observation (EWS ≥ 3) occurred within 48 hours of admission.

Key Insights and Ideas

  • Digitization Alone Isn't a Magic Bullet: The most striking finding is that simply introducing a digital EWS system, even a highly usable one, did not automatically translate into improved patient outcomes. This challenges the assumption that technological advancement alone will solve complex clinical problems.
  • Clinical Engagement is Necessary, But Not Sufficient: The study reported strong clinical engagement with the SEND system, with a good usability score. This indicates that the system was well-received by staff, but this positive reception didn't lead to better outcomes.
  • The "Work as Imagined" vs. "Work as Done" Gap: The authors highlight the potential disconnect between hospital protocols and actual clinical practice. While the digital system might remind nurses to recheck vital signs, nurses might still exercise clinical judgment and choose to deviate from the protocol, or the protocol itself might need adjustment to align with effective nursing practice.
  • Mechanisms of Impact Remain Elusive: The study struggled to pinpoint how digital EWS systems might influence outcomes. While the primary outcome didn't change, a secondary analysis hinted at a potential reduction in time to ICU admission for the entire patient population, but this was borderline and warrants further investigation. This suggests that the impact might be more nuanced than simply increased observation frequency.
  • Usability is Important, But Context Matters: The system's usability was good, but the authors suggest that the presentation of advice and the context in which it's received are crucial. If advice isn't timely or relevant, it may not effectively influence behavior.

Key Results

  • No Significant Difference in Primary Outcome: The median time to next observation (TTNO) was virtually identical between the control (128 minutes) and intervention (131 minutes) arms. The hazard ratio for TTNO was 0.99 (95% CI 0.91-1.07, P=.73), indicating no statistically significant difference.
  • No Significant Difference in Secondary Outcomes: For the primary analysis cohort, the digital EWS system showed no significant effect on time to death, length of stay, or time to unplanned ICU admission.
  • Usability Score: The system usability score was 77.6, which is considered good usability.
  • Limited Cardiac Arrest Data: Only 48 cardiac arrest calls occurred across both arms, making it insufficient to model this outcome.
  • Potential Borderline Reduction in ICU Admission (Entire Population): When analyzing the entire patient population (not just those with triggering EWS), there was a borderline reduction in time to ICU admission in the intervention arm (hazard ratio 1.25, 95% CI 1.02-1.54), though this was not statistically significant at P=.05.

Limitations and Future Work

  • Limitations:
    • Non-Randomized Clusters: The stepped-wedge design involved clusters of wards that were not randomized but determined by the rollout strategy. This could introduce bias if secular trends exist.
  • Small Number of Secondary Endpoints: For some secondary outcomes, the number of events was very low, limiting the power to detect differences.
  • Usability Survey Limitations: The usability survey was only administered at the first hospital site, and there was a low response rate, potentially leading to a non-representative sample.
  • Future Work:
    • Integrating with Clinical Pathways: The authors strongly suggest that future research should focus on how digital EWS systems can be integrated with new clinical pathways that actively adjust staff behaviors to improve patient outcomes.
  • Understanding Behavioral Mechanisms: Further investigation is needed to understand the complex mechanisms by which digital systems influence staff behavior and clinical decision-making.
  • Exploring Secondary Data Use: The benefits of using the data collected by digital systems for clinical governance and research were not examined in this study and could be a future avenue of exploration.

Summary updated

Abstract

We demonstrated strong clinical engagement with the system. We found no difference in any of the predefined patient outcomes, suggesting that the introduction of a highly usable electronic system can be achieved without impacting clinical care. Our findings contrast with previous claims that digital EWS systems are associated with improvement in clinical outcomes. Future research should investigate how digital EWS systems can be integrated with new clinical pathways adjusting staff behaviors to improve patient outcomes.
BibTeX
@article{wong2024effect,
  author = {David Wong and Timothy Bonnici and Stephen Gerry and Jacqueline Birks and Peter Watkinson},
  title = {{Effect of Digital Early Warning Scores on Hospital Vital Sign Observation Protocol Adherence: Stepped-Wedge Evaluation}},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  doi = {10.2196/46691},
}
Figure