Comparison of deep learning with regression analysis in creating predictive models for SARS-CoV-2 outcomes

Ahmed Abdulaal, Aatish Patel, Esmita Charani, Sarah Denny, Saleh A. Alqahtani, Gary Davies, Nabeela Mughal, Luke MooreResearch Square 2020 · DOI

Abstract

Background: Accurately predicting patient outcomes in Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) could aid patient management and allocation of healthcare resources. There are a variety of methods which can be used to develop prognostic models, ranging from logistic regression and survival analysis to more complex machine learning algorithms and deep learning. Despite several models having been created for SARS-CoV-2, most of these have been found to be highly susceptible to bias. We aimed to develop and compare two separate predictive models for death during admission with SARS-CoV-2.

Method: Between March 1 - April 24, 2020, 398 patients were identified with laboratory confirmed SARS-CoV-2 in a London teaching hospital. Data from electronic health records were extracted and used to create two predictive models using: 1) a Cox regression model and 2) an artificial neural network (ANN). Model performance profiles were assessed by validation, discrimination, and calibration.

Results: Both the Cox regression and ANN models achieved high accuracy (83.8%, 95% confidence interval (CI): 73.8 - 91.1 and 90.0%, 95% CI: 81.2 - 95.6, respectively). The area under the receiver operator curve (AUROC) for the ANN (92.6%, 95% CI: 91.1 - 94.1) was significantly greater than that of the Cox regression model (86.9%, 95% CI: 85.7 - 88.2), p=0.0136. Both models achieved acceptable calibration with Brier scores of 0.13 and 0.11 for the Cox model and ANN, respectively.

Conclusion: We demonstrate an ANN which is non-inferior to a Cox regression model but with potential for further development such that it can learn as new data becomes available. Deep learning techniques are particularly suited to complex datasets with non-linear solutions, which make them appropriate for use in conditions with a paucity of prior knowledge. Accurate prognostic models for SARS-CoV-2 can provide benefits at the patient, departmental and organisational level.

Introduction

During the initial surge of the COVID-19 pandemic, healthcare systems worldwide faced unprecedented pressure, creating a desperate need for objective tools to assist in resource allocation and patient management. While numerous prognostic models were rapidly developed, many were limited by small sample sizes, susceptibility to bias, or reliance on subjective clinical predictors. This study aimed to address these gaps by comparing traditional statistical methods with modern deep learning techniques to predict mortality risk at the point of hospital admission using objective data available during a patient's first contact with healthcare workers.

Methods

The researchers conducted a retrospective study using Electronic Health Records (EHR) from a teaching hospital in West London. The cohort included 398 patients with laboratory-confirmed SARS-CoV-2 admitted between March 1 and April 24, 2020.

Two distinct predictive models were developed and compared:

  1. Cox Proportional Hazards Regression: A standard statistical approach for modeling time-to-event data. A parsimonious model was developed using backwards elimination to identify the most significant predictors.
  2. Artificial Neural Network (ANN): A deep learning model built with the TensorFlow library. The architecture featured an input layer for 21 predictors, two hidden layers (8 and 4 units), and a 20% dropout rate to mitigate overfitting.

The models utilized 21 objective predictors available at admission, including demographics (age, sex), comorbidities (diabetes, heart disease, obesity), and presenting symptoms (fever, cough, confusion). Predictors with high rates of missing data, such as smoking history and ethnicity, were excluded to maintain data integrity. Both models were developed in accordance with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis) guidelines to ensure scientific rigor.

Results

The comparative analysis revealed that the deep learning approach provided superior predictive performance. The ANN achieved an Area Under the Receiver Operating Curve (AUROC) of 92.6% (95% CI: 91.1 - 94.1), which was significantly higher than the 86.9% (95% CI: 85.7 - 88.2) achieved by the Cox regression model (p=0.0136).

In terms of overall accuracy, the ANN reached 90.0% compared to 83.8% for the Cox model. Crucially, the ANN demonstrated higher sensitivity (64.7% vs. 50.0%), indicating it was more effective at identifying high-risk patients who ultimately succumbed to the virus. Both models showed acceptable calibration, with Brier scores of 0.11 for the ANN and 0.13 for the Cox model (where lower scores indicate better performance).

Discussion

The superior performance of the ANN suggests that the clinical progression of COVID-19 involves complex, non-linear interactions between risk factors that traditional linear models may fail to capture. For instance, the interaction between age and specific comorbidities may not be additive, a nuance that deep learning architectures are specifically designed to handle.

Despite the differences in performance, both models identified confusion, collapse, and dyspnoea as critical predictors of mortality. This consistency between "black box" AI and traditional statistics provides clinical validation for the models' logic. Interestingly, abdominal pain appeared as a protective factor in the Cox model but had negligible impact in the ANN; the authors suggest the ANN more gracefully handled this nuance, potentially recognizing it as a marker for a milder gastrointestinal-focused phenotype of the disease.

A significant advantage of the ANN approach is its capacity for incremental or "active" learning. Unlike static regression models, ANNs can be updated as new data becomes available, allowing the model to adapt to emerging variants or changes in treatment protocols.

Conclusion

This study demonstrates that deep learning techniques like ANNs offer a powerful alternative to traditional regression for prognostic modeling in emerging infectious diseases. While limited by its single-center design and the omission of certain variables due to data quality, the model provides a foundation for more advanced tools. Future work should focus on multi-center validation and the integration of longitudinal data using Recurrent Neural Networks (RNNs) to provide dynamic risk assessments throughout a patient's stay. Accurate prognostic models for SARS-CoV-2 can provide significant benefits at the patient, departmental, and organizational levels.

Summary updated

Abstract

<title>Abstract</title> Background Accurately predicting patient outcomes in Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) could aid patient management and allocation of healthcare resources. There are a variety of methods which can be used to develop prognostic models, ranging from logistic regression and survival analysis to more complex machine learning algorithms and deep learning. Despite several models having been created for SARS-CoV-2, most of these have been found to be highly susceptible to bias. We aimed to develop and compare two separate predictive models for death during admission with SARS-CoV-2.MethodBetween March 1 - April 24, 2020, 398 patients were identified with laboratory confirmed SARS-CoV-2 in a London teaching hospital. Data from electronic health records were extracted and used to create two predictive models using: 1) a Cox regression model and 2) an artificial neural network (ANN). Model performance profiles were assessed by validation, discrimination, and calibration.Results Both the Cox regression and ANN models achieved high accuracy (83.8%, 95% confidence interval (CI): 73.8 - 91.1 and 90.0%, 95% CI: 81.2 - 95.6, respectively). The area under the receiver operator curve (AUROC) for the ANN (92.6%, 95% CI: 91.1 - 94.1) was significantly greater than that of the Cox regression model (86.9%, 95% CI: 85.7 - 88.2), p=0.0136. Both models achieved acceptable calibration with Brier scores of 0.13 and 0.11 for the Cox model and ANN, respectively. ConclusionWe demonstrate an ANN which is non-inferior to a Cox regression model but with potential for further development such that it can learn as new data becomes available. Deep learning techniques are particularly suited to complex datasets with non-linear solutions, which make them appropriate for use in conditions with a paucity of prior knowledge. Accurate prognostic models for SARS-CoV-2 can provide benefits at the patient, departmental and organisational level.
BibTeX
@inproceedings{abdulaal2020comparison,
  author = {Ahmed Abdulaal and Aatish Patel and Esmita Charani and Sarah Denny and Saleh A. Alqahtani and Gary Davies and Nabeela Mughal and Luke Moore},
  title = {{Comparison of deep learning with regression analysis in creating predictive models for SARS-CoV-2 outcomes}},
  booktitle = {Research Square},
  year = {2020},
  doi = {10.21203/rs.3.rs-52481/v4},
}
Figure