Abstract
Introduction: The Epic Deterioration Index (EDI) is a proprietary prediction model implemented in over 100 U.S. hospitals and was widely used to support medical decision-making during the COVID-19 pandemic. The EDI has not been independently evaluated, and other proprietary models have been shown to be biased against vulnerable populations. Methods: We studied adult patients admitted with COVID-19 to non-ICU care at a large academic medical center from March 9 through May 20, 2020. We used the EDI, calculated at 15-minute intervals, to predict a composite outcome of ICU-level care, mechanical ventilation, or in-hospital death. Results: Among 392 COVID-19 hospitalizations, 103 (26%) met the composite outcome. Area under the receiver-operating-characteristic curve (AUC) of the EDI was 0.79 (95% CI 0.74-0.84). EDI predictions did not differ by race or sex. A threshold of 68.8 identified a high-risk group (74% probability of deterioration) with a median lead time of 24 hours. Conclusion: The EDI identifies small subsets of high- and low-risk COVID-19 patients with fair discrimination and no evidence of bias by race or sex, highlighting the importance of independent evaluation of proprietary models.
Introduction: The "Black Box" of Clinical Decision Support
In modern healthcare, "Early Warning Systems" (EWS) function as digital smoke detectors. By ingesting real-time streams of patient data—including vital signs, laboratory results, and nursing assessments—these algorithms aim to predict which patients are at risk of clinical deterioration, or "crashing," before the event occurs.
The Epic Deterioration Index (EDI) is one of the most widely deployed EWS tools in the United States. However, because the EDI is a proprietary product of Epic Systems, its internal logic and variable weightings remain a secret. Despite its widespread use during the COVID-19 pandemic, the model lacked peer-reviewed, independent validation for this specific patient population. Furthermore, given that previous research has identified significant racial bias in other proprietary healthcare algorithms, there was an urgent need to evaluate whether the EDI was both accurate and equitable.
Study Design and Methodology
Researchers at Michigan Medicine conducted a retrospective cohort study to pull back the curtain on the EDI’s performance. The study analyzed 392 hospitalizations (representing 369 unique patients) of adults admitted to non-ICU beds with COVID-19 between March and May 2020.
Key Evaluation Metrics
- Predictive Target: The study focused on a composite outcome defined as the need for ICU-level care, mechanical ventilation, or in-hospital death.
- Discrimination: Performance was measured using the Area Under the Receiver-Operating-Characteristic curve (AUC), which assesses how well the model distinguishes between patients who will deteriorate and those who will remain stable.
- Threshold Analysis: The team identified specific "clinically actionable" scores to determine how the model might be used to trigger interventions or identify candidates for early discharge.
- Bias Testing: A disparate impact analysis was performed, stratifying results by age, sex, and race to ensure the model did not perform poorly for specific demographic groups.
Results: Fair Discrimination and Actionable Lead Times
The EDI demonstrated an AUC of 0.79, indicating "fair" to "good" discrimination. This performance is comparable to other established, non-proprietary indices like the Rothman Index or eCART.
Clinical Utility and Thresholds
The study found that the EDI is most effective when clinicians look for specific thresholds rather than single snapshots of data. Because the score is updated every 15 minutes, it can fluctuate; however, crossing specific high-risk boundaries proved highly predictive:
- High-Risk Threshold (68.8): Patients who met or exceeded this score represented 14% of the cohort and had a 74% probability of experiencing the composite outcome. Crucially, the model provided a median lead time of 24 hours before the event, offering a significant window for clinical intervention.
- Low-Risk Threshold (37.9): For patients who remained stable for at least 48 hours, this threshold yielded a 90% Negative Predictive Value (NPV), suggesting it could help identify patients safe for potential discharge.
Algorithmic Fairness
Significantly, the researchers found no evidence of demographic bias. The EDI performed consistently across race and sex, a vital finding given the diversity of the cohort (43% African American and 43% women) and the history of bias in other clinical algorithms.
Discussion and Limitations
While the EDI proved to be a useful tool, the study noted a tendency for the model to overpredict risk. When the model assigned a high score, the actual probability of deterioration was often lower than the score itself might suggest if interpreted as a direct percentage.
Furthermore, the study was limited to a single academic medical center. The Michigan Medicine cohort had a higher prevalence of comorbidities (such as chronic kidney disease and hypertension) than other major COVID-19 datasets, which may influence how the model generalizes to other settings. The sample size of 392 is also small compared to the 130,000+ hospitalizations used by Epic to originally develop the model.
Conclusion
This independent validation suggests that the Epic Deterioration Index is a capable tool for identifying high-risk COVID-19 patients with a meaningful lead time. However, its proprietary nature remains a challenge for transparency and clinical trust. The authors advocate for continued independent evaluation of "black box" models and suggest that future research should compare proprietary tools directly against open-source alternatives to determine if secret algorithms provide any tangible clinical advantage over transparent ones.
Summary updated