Machine-Learning prognostic models from the 2014-16 Ebola outbreak: data-harmonization challenges, validation strategies, and mHealth applications

Andrés Colubri, Mary‐Anne Hartley, Mathew Siakor, Vanessa Wolfman, Tom Sesay, August Felix, Adam C. Levine, Pardis C. SabetibioRxiv 2018 · DOI

TL;DR

This paper presents machine learning models to predict Ebola patient outcomes, developed from a large, harmonized dataset. These models, integrated into a mobile app, aim to improve clinical decision-making in resource-limited settings, demonstrating that ML can effectively capture clinical intuition and provide actionable guidance.

What Problem Does This Paper Address?

The 2014-2016 Ebola outbreak highlighted the critical need for field-deployable tools to manage the disease, especially in resource-scarce environments. Existing clinical guidelines, while valuable, are often difficult to access and use on mobile devices. Furthermore, the heterogeneity of data collected across different treatment centers during outbreaks poses a significant challenge for developing generalizable predictive models. This paper tackles these issues by developing and validating machine learning prognostic models for Ebola, harmonizing diverse datasets, and integrating them into an mHealth application to support frontline healthcare workers.

Methods

The authors employed multivariate logistic regression to build prognostic models. A key methodological challenge was data harmonization, where data from different Ebola Treatment Units (ETUs) with varying collection protocols were standardized. They used techniques like intra-site normalization for PCR data (CT values) and multiple imputation to handle missing values, particularly for variables like body temperature. Variable selection was performed using penalized logistic regression (Elastic Net) to identify the most predictive features. The models were then internally validated using bootstrap resampling and externally validated on independent datasets.

Datasets

The primary dataset used for training and internal validation comprised 470 confirmed EVD cases collected from five ETUs operated by International Medical Corps (IMC) in Sierra Leone and Liberia between September 2014 and September 2015. This is noted as the largest and most diverse clinical EVD dataset available at the time.

For external validation, two independent datasets were used:

  1. KGH dataset: 106 Ebola-positive patients treated at Kenema Government Hospital (KGH) in Sierra Leone during the early months of the outbreak.
  2. GOAL dataset: 158 Ebola patients treated at a GOAL Global ETU in Port Loko, Sierra Leone, during the later months of the epidemic.

Key Insights and Ideas

  • ML can approximate clinical intuition: The study found that machine learning models, when properly designed and applied to rich data, can approximate the "gestalt" or clinical intuition that experienced physicians develop. This is crucial for situations where expert experience is limited.
  • Data harmonization is critical for generalizability: The success in building models that performed well across different sites and time points underscores the importance of robust data harmonization techniques in outbreak settings.
  • mHealth as a powerful delivery mechanism: Integrating prognostic models into a mobile app makes complex clinical knowledge actionable and accessible to frontline healthcare workers, especially in remote areas.
  • Predictors of mortality: Viral load and patient age emerged as the strongest predictors of death, but a parsimonious model including clinical signs and symptoms also showed good performance.
  • Wellness scale as a valuable predictor: Incorporating an observational "wellness scale" (WS) into models proved to be at least as effective, and sometimes superior, to models relying solely on detailed clinical features.

Key Results

  • The developed prognostic models achieved an Area Under the Curve (AUC) ranging from 0.7 to 0.8, with overall accuracy between 64% and 74%.
  • The parsimonious model, incorporating viral load, age, body temperature, bleeding, jaundice, dyspnea, dysphagia, and referral time, performed comparably to observational assessments by experienced health workers.
  • External validation on the GOAL dataset showed AUCs of 0.82, 0.84, 0.73, and 0.82 for the parsimonious, parsimonious without temperature, clinical-only, and minimal models, respectively.
  • The "Ebola Care Guidelines" app was developed, integrating these models with evidence-based care recommendations, designed for offline use after initial installation.

Limitations and Future Work

  • Data limitations: Despite being the largest study of its kind, the amount and quality of available clinical data remain limited. Variations in clinical assessments, data collection errors, and differences in lab protocols can still affect data accuracy.
  • Generalizability: While efforts were made to harmonize data and validate models externally, achieving truly generalizable models across all contexts remains a challenge. Future work should focus on larger and better datasets with consensus mechanisms for data consistency.
  • Model updates: The authors envision a future where apps can retrieve updated models from a remote server, eliminating the need for app updates and creating a positive feedback loop where improved data generation leads to more precise models.
  • Broader application: The approach is generalizable and can be applied to develop similar mHealth tools for other tropical diseases in rural and low-resource areas.

Summary updated

Abstract

Abstract Background We created a family of prognostic models for Ebola virus disease from the largest dataset of EVD patients published to date. We incorporated these models into an app, “Ebola Care Guidelines”, that provides access to recommended, evidence-based supportive care guidelines and highlights the signs/symptoms with the largest contribution to prognosis. Methods We applied multivariate logistic regression on 470 patients admitted to five Ebola treatment units in Liberia and Sierra Leone during the 2014-16 outbreak. We validated the models with two independent datasets from Sierra Leone. Findings Viral load and age were the most important predictors of death. We generated a parsimonious model including viral load, age, body temperature, bleeding, jaundice, dyspnea, dysphagia, and referral time recorded at triage. We also constructed fallback models for when variables in the parsimonious model are unavailable. The performance of the parsimonious model approached the predictive power of observational wellness assessments by experienced health workers, with Area Under the Curve (AUC) ranging from 0.7 to 0.8 and overall accuracy of 64% to 74%. Interpretation Machine-learning models and mHealth tools have the potential for improving the standard of care in low-resource settings and emergency scenarios, but data incompleteness and lack of generalizable models are major obstacles. We showed how harmonization of multiple datasets yields prognostic models that can be validated across different cohorts. Similar performance between the parsimonious model and those incorporating expert wellness assessments suggests that clinically-guided machine learning approaches can recapitulate clinical expertise, and thus be useful when such expertise is unavailable. We also demonstrated with our guidelines app how integration of those models with mobile technologies enables deployable clinical management support tools that facilitate access to comprehensive bodies of medical knowledge. Funding Howard Hughes Medical Institute, US National Institutes of Health
BibTeX
@misc{colubri2018machine,
  author = {Andrés Colubri and Mary‐Anne Hartley and Mathew Siakor and Vanessa Wolfman and Tom Sesay and August Felix and Adam C. Levine and Pardis C. Sabeti},
  title = {{Machine-Learning prognostic models from the 2014-16 Ebola outbreak: data-harmonization challenges, validation strategies, and mHealth applications}},
  year = {2018},
  doi = {10.1101/294587},
  howpublished = {bioRxiv},
}
Figure