Prognostic Modeling of COVID-19 Using Artificial Intelligence in the United Kingdom: Model Development and Validation (Preprint)

Ahmed Abdulaal, Aatish Patel, Esmita Charani, Sarah Denny, Nabeela Mughal, Luke Moore2020 · DOI

TL;DR

This paper introduces an Artificial Neural Network (ANN) model that predicts the mortality risk of COVID-19 patients at the point of admission. The model achieves high accuracy and provides interpretable insights into the key factors influencing mortality, demonstrating the potential of deep learning for rapid pandemic response.

What Problem Does This Paper Address?

The COVID-19 pandemic presented a significant public health challenge, characterized by a high case fatality rate, particularly in the early stages. While some general predictors of outcome were emerging, there was a critical lack of validated, patient-specific prognostic models or scoring systems that could inform clinical decisions at the earliest possible moment. This gap made it difficult for healthcare providers to anticipate patient deterioration, manage hospital resources effectively (like ICU beds), and have informed discussions with patients and their families.

Methods

The researchers developed and validated an Artificial Neural Network (ANN) model. The ANN was designed to take a set of patient features as input and output a patient-specific mortality risk prediction at the point of hospital admission. The model was trained and validated using data from patients admitted to a West London teaching hospital with a confirmed SARS-CoV-2 infection.

Key methodological aspects include:

  • Feature Engineering: The ANN utilized a comprehensive set of patient features, including demographics (age, gender), comorbidities (hypertension, diabetes, respiratory diseases, etc.), smoking history, and presenting symptoms (cough, fever, dyspnea, etc.).
  • ANN Architecture: The model employed a multi-layered ANN with densely connected hidden layers, utilizing rectifier activation functions. The output layer used sigmoid activation to provide a probability of death.
  • Training and Validation: The data was split into training (80%) and testing (20%) sets. K-fold cross-validation (10 folds) was used to prevent overfitting and calculate robust performance metrics.
  • Interpretability: The study incorporated SHAP (Shapley Additive Explanations) values to understand the relative importance of different clinical attributes in predicting mortality, both on a patient-specific basis and overall. This addresses a common challenge with deep learning models, which can sometimes be "black boxes."

Datasets

The study utilized a dataset of 398 patients admitted to a West London teaching hospital with a positive RT-PCR test for SARS-CoV-2 between February 2, 2020, and April 22, 2020. This dataset comprised 11,144 data points encompassing 22 input features. The data was extracted from electronic health records (EHRs) and anonymized.

Key Insights and Ideas

  • Early, Patient-Specific Risk Stratification: The core innovation is the development of a model that provides a mortality risk score at the point of admission. This allows for immediate clinical decision-making and resource allocation, which is crucial in a rapidly evolving pandemic.
  • Deep Learning's Utility in Pandemics: The study highlights the power of deep learning approaches to rapidly develop prognostic models in situations where established scoring systems are lacking, as was the case with COVID-19.
  • Explainable AI for Clinical Trust: By using SHAP values, the model not only predicts risk but also explains why it makes that prediction. This interpretability is vital for clinician adoption and trust in AI-driven tools.
  • Key Mortality Predictors Identified: The model identified altered mentation, dyspnea, and increasing age as the most significant predictors of mortality. This aligns with existing literature but provides a validated, data-driven confirmation within this specific cohort.
  • Smoking History Nuance: Interestingly, smoking history was found to be a less significant predictor than might be expected. The authors suggest this could be due to how smoking status was encoded (grouping current and ex-smokers) and potentially protective effects of nicotine, a finding that warrants further investigation.

Key Results

The ANN model demonstrated strong performance in predicting patient-specific mortality:

  • Overall Accuracy: 86.25% on the test set.
  • Sensitivity: 87.50% (correctly identifying those who would die).
  • Specificity: 85.94% (correctly identifying those who would survive).
  • Negative Predictive Value (NPV): 96.49% (high confidence that patients predicted to survive will indeed survive).
  • Area Under the Receiver Operating Characteristic Curve (AUROC): 90.12%, indicating excellent discriminative ability.
  • Feature Importance: Altered mentation, new dyspnea, and increasing age were identified as the most significant predictors of mortality. Moderate predictors included collapse, male gender, new cough, and known respiratory pathology.

Limitations and Future Work

The authors acknowledge several limitations:

  • Single-Site Data: The model was trained on data from a single hospital site. Generalizing these results to other populations or healthcare settings requires validation with larger, multi-site datasets.
  • Unaccounted Factors: The model did not include hematological, biochemical, radiological, or microbiological data, which could further enhance predictive power. The ANN architecture is amenable to incorporating these additional features.
  • Censored Patients: A small percentage of patients were still hospitalized at the end of the study period. While considered "alive," their ultimate outcomes might differ.
  • Duration of Prediction: The current model is designed for point-of-admission prediction. The authors suggest that more complex models like Recurrent Neural Networks (RNNs) could be used for real-time outcome predictions throughout an admission.

Future work could involve expanding the model to include more diverse data types, validating it across multiple institutions, and exploring RNNs for continuous monitoring and prediction.

Summary updated

Abstract

<sec> <title>BACKGROUND</title> The current severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) outbreak is a public health emergency and the case fatality rate in the United Kingdom is significant. Although there appear to be several early predictors of outcome, there are no currently validated prognostic models or scoring systems applicable specifically to patients with confirmed SARS-CoV-2. </sec> <sec> <title>OBJECTIVE</title> We aim to create a point-of-admission mortality risk scoring system using an artificial neural network (ANN). </sec> <sec> <title>METHODS</title> We present an ANN that can provide a patient-specific, point-of-admission mortality risk prediction to inform clinical management decisions at the earliest opportunity. The ANN analyzes a set of patient features including demographics, comorbidities, smoking history, and presenting symptoms and predicts patient-specific mortality risk during the current hospital admission. The model was trained and validated on data extracted from 398 patients admitted to hospital with a positive real-time reverse transcription polymerase chain reaction (RT-PCR) test for SARS-CoV-2. </sec> <sec> <title>RESULTS</title> Patient-specific mortality was predicted with 86.25% accuracy, with a sensitivity of 87.50% (95% CI 61.65%-98.45%) and specificity of 85.94% (95% CI 74.98%-93.36%). The positive predictive value was 60.87% (95% CI 45.23%-74.56%), and the negative predictive value was 96.49% (95% CI 88.23%-99.02%). The area under the receiver operating characteristic curve was 90.12%. </sec> <sec> <title>CONCLUSIONS</title> This analysis demonstrates an adaptive ANN trained on data at a single site, which demonstrates the early utility of deep learning approaches in a rapidly evolving pandemic with no established or validated prognostic scoring systems. </sec>
BibTeX
@misc{abdulaal2020prognostic,
  author = {Ahmed Abdulaal and Aatish Patel and Esmita Charani and Sarah Denny and Nabeela Mughal and Luke Moore},
  title = {{Prognostic Modeling of COVID-19 Using Artificial Intelligence in the United Kingdom: Model Development and Validation (Preprint)}},
  year = {2020},
  doi = {10.2196/preprints.20259},
}
Figure