Evaluating a Widely Implemented Proprietary Deterioration Index Model among Hospitalized Patients with COVID-19

Karandeep Singh, Thomas S. Valley, Shengpu Tang, Benjamin Y. Li, Fahad Kamran, Michael W. Sjoding, Jenna Wiens, Erkin Ötleş, John P. Donnelly, Melissa Y. Wei, Jonathon McBride, Jie Cao, Carleen Penoza, John Z. Ayanian, Brahmajee K. NallamothuAnnals 2020 · DOI

Abstract

Introduction: The Epic Deterioration Index (EDI) is a proprietary prediction model implemented in over 100 U.S. hospitals and was widely used to support medical decision-making during the COVID-19 pandemic. The EDI has not been independently evaluated, and other proprietary models have been shown to be biased against vulnerable populations. Methods: We studied adult patients admitted with COVID-19 to non-ICU care at a large academic medical center from March 9 through May 20, 2020. We used the EDI, calculated at 15-minute intervals, to predict a composite outcome of ICU-level care, mechanical ventilation, or in-hospital death. Results: Among 392 COVID-19 hospitalizations, 103 (26%) met the composite outcome. Area under the receiver-operating-characteristic curve (AUC) of the EDI was 0.79 (95% CI 0.74-0.84). EDI predictions did not differ by race or sex. A threshold of 68.8 identified a high-risk group (74% probability of deterioration) with a median lead time of 24 hours. Conclusion: The EDI identifies small subsets of high- and low-risk COVID-19 patients with fair discrimination and no evidence of bias by race or sex, highlighting the importance of independent evaluation of proprietary models.

Introduction: The "Black Box" of Clinical Decision Support

In modern healthcare, "Early Warning Systems" (EWS) function as digital smoke detectors. By ingesting real-time streams of patient data—including vital signs, laboratory results, and nursing assessments—these algorithms aim to predict which patients are at risk of clinical deterioration, or "crashing," before the event occurs.

The Epic Deterioration Index (EDI) is one of the most widely deployed EWS tools in the United States. However, because the EDI is a proprietary product of Epic Systems, its internal logic and variable weightings remain a secret. Despite its widespread use during the COVID-19 pandemic, the model lacked peer-reviewed, independent validation for this specific patient population. Furthermore, given that previous research has identified significant racial bias in other proprietary healthcare algorithms, there was an urgent need to evaluate whether the EDI was both accurate and equitable.

Study Design and Methodology

Researchers at Michigan Medicine conducted a retrospective cohort study to pull back the curtain on the EDI’s performance. The study analyzed 392 hospitalizations (representing 369 unique patients) of adults admitted to non-ICU beds with COVID-19 between March and May 2020.

Key Evaluation Metrics

  • Predictive Target: The study focused on a composite outcome defined as the need for ICU-level care, mechanical ventilation, or in-hospital death.
  • Discrimination: Performance was measured using the Area Under the Receiver-Operating-Characteristic curve (AUC), which assesses how well the model distinguishes between patients who will deteriorate and those who will remain stable.
  • Threshold Analysis: The team identified specific "clinically actionable" scores to determine how the model might be used to trigger interventions or identify candidates for early discharge.
  • Bias Testing: A disparate impact analysis was performed, stratifying results by age, sex, and race to ensure the model did not perform poorly for specific demographic groups.

Results: Fair Discrimination and Actionable Lead Times

The EDI demonstrated an AUC of 0.79, indicating "fair" to "good" discrimination. This performance is comparable to other established, non-proprietary indices like the Rothman Index or eCART.

Clinical Utility and Thresholds

The study found that the EDI is most effective when clinicians look for specific thresholds rather than single snapshots of data. Because the score is updated every 15 minutes, it can fluctuate; however, crossing specific high-risk boundaries proved highly predictive:

  • High-Risk Threshold (68.8): Patients who met or exceeded this score represented 14% of the cohort and had a 74% probability of experiencing the composite outcome. Crucially, the model provided a median lead time of 24 hours before the event, offering a significant window for clinical intervention.
  • Low-Risk Threshold (37.9): For patients who remained stable for at least 48 hours, this threshold yielded a 90% Negative Predictive Value (NPV), suggesting it could help identify patients safe for potential discharge.

Algorithmic Fairness

Significantly, the researchers found no evidence of demographic bias. The EDI performed consistently across race and sex, a vital finding given the diversity of the cohort (43% African American and 43% women) and the history of bias in other clinical algorithms.

Discussion and Limitations

While the EDI proved to be a useful tool, the study noted a tendency for the model to overpredict risk. When the model assigned a high score, the actual probability of deterioration was often lower than the score itself might suggest if interpreted as a direct percentage.

Furthermore, the study was limited to a single academic medical center. The Michigan Medicine cohort had a higher prevalence of comorbidities (such as chronic kidney disease and hypertension) than other major COVID-19 datasets, which may influence how the model generalizes to other settings. The sample size of 392 is also small compared to the 130,000+ hospitalizations used by Epic to originally develop the model.

Conclusion

This independent validation suggests that the Epic Deterioration Index is a capable tool for identifying high-risk COVID-19 patients with a meaningful lead time. However, its proprietary nature remains a challenge for transparency and clinical trust. The authors advocate for continued independent evaluation of "black box" models and suggest that future research should compare proprietary tools directly against open-source alternatives to determine if secret algorithms provide any tangible clinical advantage over transparent ones.

Summary updated

Abstract

<b>Rationale:</b> The Epic Deterioration Index (EDI) is a proprietary prediction model implemented in over 100 U.S. hospitals that was widely used to support medical decision-making during the coronavirus disease (COVID-19) pandemic. The EDI has not been independently evaluated, and other proprietary models have been shown to be biased against vulnerable populations. <b>Objectives:</b> To independently evaluate the EDI in hospitalized patients with COVID-19 overall and in disproportionately affected subgroups. <b>Methods:</b> We studied adult patients admitted with COVID-19 to units other than the intensive care unit at a large academic medical center from March 9 through May 20, 2020. We used the EDI, calculated at 15-minute intervals, to predict a composite outcome of intensive care unit-level care, mechanical ventilation, or in-hospital death. In a subset of patients hospitalized for at least 48 hours, we also evaluated the ability of the EDI to identify patients at low risk of experiencing this composite outcome during their remaining hospitalization. <b>Results:</b> Among 392 COVID-19 hospitalizations meeting inclusion criteria, 103 (26%) met the composite outcome. The median age of the cohort was 64 (interquartile range, 53-75) with 168 (43%) Black patients and 169 (43%) women. The area under the receiver-operating characteristic curve of the EDI was 0.79 (95% confidence interval, 0.74-0.84). EDI predictions did not differ by race or sex. When exploring clinically relevant thresholds of the EDI, we found patients who met or exceeded an EDI of 68.8 made up 14% of the study cohort and had a 74% probability of experiencing the composite outcome during their hospitalization with a sensitivity of 39% and a median lead time of 24 hours from when this threshold was first exceeded. Among the 286 patients hospitalized for at least 48 hours who had not experienced the composite outcome, 14 (13%) never exceeded an EDI of 37.9, with a negative predictive value of 90% and a sensitivity above this threshold of 91%. <b>Conclusions:</b> We found the EDI identifies small subsets of high-risk and low-risk patients with COVID-19 with good discrimination, although its clinical use as an early warning system is limited by low sensitivity. These findings highlight the importance of independent evaluation of proprietary models before widespread operational use among patients with COVID-19.
BibTeX
@article{singh2020evaluating,
  author = {Karandeep Singh and Thomas S. Valley and Shengpu Tang and Benjamin Y. Li and Fahad Kamran and Michael W. Sjoding and Jenna Wiens and Erkin Ötleş and John P. Donnelly and Melissa Y. Wei and Jonathon McBride and Jie Cao and Carleen Penoza and John Z. Ayanian and Brahmajee K. Nallamothu},
  title = {{Evaluating a Widely Implemented Proprietary Deterioration Index Model among Hospitalized Patients with COVID-19}},
  journal = {Annals},
  year = {2020},
  doi = {10.1513/annalsats.202006-698oc},
}

Loading related content...

Figure