A multistage multimodal deep learning model for disease severity assessment and early warnings of high-risk patients of COVID-19

Zhuo Li, Ruiqing Xu, Yifei Shen, Jiannong Cao, Ben Wang, Ying Zhang, Shikang LiFrontiers in Public Health 2022 · DOI

Abstract

The outbreak of coronavirus disease 2019 (COVID-19) has caused massive infections and large death tolls worldwide. Despite many studies on the clinical characteristics and the treatment plans of COVID-19, they rarely conduct in-depth prognostic research on leveraging consecutive rounds of multimodal clinical examination and laboratory test data to facilitate clinical decision-making for the treatment of COVID-19. To address this issue, we propose a multistage multimodal deep learning (MMDL) model to (1) first assess the patient's current condition (i.e., the mild and severe symptoms), then (2) give early warnings to patients with mild symptoms who are at high risk to develop severe illness. In MMDL, we build a sequential stage-wise learning architecture whose design philosophy embodies the model's predicted outcome and does not only depend on the current situation but also the history. Concretely, we meticulously combine the latest round of multimodal clinical data and the decayed past information to make assessments and predictions. Experimental results validate our assumption that sequential stage-wise learning outperforms single-stage learning, and that MMDL with multimodal inputs can beat any reduced model with single-modal inputs only.

Introduction

Clinical management of COVID-19 presents a significant triage challenge. While the majority of patients experience mild symptoms, a critical subset progresses rapidly to severe illness, necessitating intensive care and placing immense strain on medical infrastructure. Traditional AI models often suffer from data fragmentation—relying on a single snapshot or a single data type—which fails to capture the dynamic "big picture" of a patient's health trajectory.

The proposed Multistage Multimodal Deep Learning (MMDL) model addresses these gaps by treating the clinical journey as a sequence of multimodal data points. This allows the system to learn from the history of disease progression rather than just its instantaneous state, providing both current severity assessments and early warnings for potential deterioration.

Methods and Architecture

The MMDL model is built upon two primary innovations: a hierarchical feature extractor and a sequential stage-wise learning framework.

Two-Layer Multimodal Feature Extractor

To handle the diversity of clinical data, the model employs a two-layer extraction process:

  1. Intra-modal Hierarchy: Separate Multi-Layer Perceptron (MLP) networks extract latent features from each of the 13 modalities independently.
  2. Cross-modal Hierarchy: These independent features are concatenated and processed through another MLP to identify complex correlations between different data types, such as the relationship between inflammatory markers and renal function changes.

Sequential Stage-Wise Learning

Standard recurrent architectures like LSTMs often require long sequences to be effective. However, clinical data for COVID-19 patients is typically sparse, often consisting of only a few rounds of tests. MMDL utilizes a custom stage-wise architecture with a decay factor ($\alpha$). This mathematical approach allows the model to incorporate historical data while prioritizing the most recent clinical rounds, reflecting the reality that the most recent 48–72 hours are the most predictive of immediate outcomes.

Dataset and Patient Characteristics

The study utilized a clinical dataset of 216 COVID-19 patients from the Chongqing Public Health Medical Center. The cohort included 186 mild cases and 30 severe cases. The data encompassed 13 distinct modalities, including demographics, clinical manifestations, and 11 categories of laboratory tests (e.g., blood lipids, electrolytes, liver and renal function).

Demographics of the patient cohort, illustrating the distribution of disease severity, gender, age, and the time elapsed from symptom onset to hospital admission.

Clinical manifestations played a vital role in the initial assessment. Symptoms such as cough and fever were prevalent across the cohort, but their intensity and combination with other markers provided the necessary signals for the MMDL model to differentiate between mild and severe progression.

Comparison of clinical manifestations between mild and severe patients, highlighting the frequency of symptoms like cough, fever, and shortness of breath.

Key Results and Insights

The experiments confirmed that incorporating temporal history significantly enhances predictive power. Severity assessment accuracy improved from 96.26% (single-stage) to 98.10% when five consecutive rounds of data were used. For the high-risk "severe" group, the F1 score saw a substantial jump from 0.8091 to 0.9436.

A critical finding was the "Power of Multimodality." The full MMDL model consistently outperformed "reduced" versions that used only single modalities. Specifically, the fusion of inflammation markers, liver function, and blood lipids emerged as the most potent discriminator for disease severity.

While the model excelled at immediate and short-term forecasting, its performance in long-term early warning (predicting transitions up to four stages ahead) showed a decrease in F1 scores, highlighting the inherent complexity of long-term medical prognosis.

Conclusion

The MMDL model demonstrates that a holistic, temporal approach to clinical data can significantly improve the accuracy of COVID-19 severity assessments. By integrating diverse data modalities and accounting for the disease's progression over time, the model provides a robust tool for clinical decision-making. Future research will focus on scaling this framework to other chronic conditions that require continuous multimodal monitoring.

Summary updated

Abstract

The outbreak of coronavirus disease 2019 (COVID-19) has caused massive infections and large death tolls worldwide. Despite many studies on the clinical characteristics and the treatment plans of COVID-19, they rarely conduct in-depth prognostic research on leveraging consecutive rounds of multimodal clinical examination and laboratory test data to facilitate clinical decision-making for the treatment of COVID-19. To address this issue, we propose a multistage multimodal deep learning (MMDL) model to (1) first assess the patient's current condition (i.e., the mild and severe symptoms), then (2) give early warnings to patients with mild symptoms who are at high risk to develop severe illness. In MMDL, we build a sequential stage-wise learning architecture whose design philosophy embodies the model's predicted outcome and does not only depend on the current situation but also the history. Concretely, we meticulously combine the latest round of multimodal clinical data and the decayed past information to make assessments and predictions. In each round (stage), we design a two-layer multimodal feature extractor to extract the latent feature representation across different modalities of clinical data, including patient demographics, clinical manifestation, and 11 modalities of laboratory test results. We conduct experiments on a clinical dataset consisting of 216 COVID-19 patients that have passed the ethical review of the medical ethics committee. Experimental results validate our assumption that sequential stage-wise learning outperforms single-stage learning, but history long ago has little influence on the learning outcome. Also, comparison tests show the advantage of multimodal learning. MMDL with multimodal inputs can beat any reduced model with single-modal inputs only. In addition, we have deployed the prototype of MMDL in a hospital for clinical comparison tests and to assist doctors in clinical diagnosis.
BibTeX
@inproceedings{li2022multistage,
  author = {Zhuo Li and Ruiqing Xu and Yifei Shen and Jiannong Cao and Ben Wang and Ying Zhang and Shikang Li},
  title = {{A multistage multimodal deep learning model for disease severity assessment and early warnings of high-risk patients of COVID-19}},
  booktitle = {Frontiers in Public Health},
  year = {2022},
  doi = {10.3389/fpubh.2022.982289},
}
Figure