Abstract
The outbreak of coronavirus disease 2019 (COVID-19) has caused massive infections and large death tolls worldwide. Despite many studies on the clinical characteristics and the treatment plans of COVID-19, they rarely conduct in-depth prognostic research on leveraging consecutive rounds of multimodal clinical examination and laboratory test data to facilitate clinical decision-making for the treatment of COVID-19. To address this issue, we propose a multistage multimodal deep learning (MMDL) model to (1) first assess the patient's current condition (i.e., the mild and severe symptoms), then (2) give early warnings to patients with mild symptoms who are at high risk to develop severe illness. In MMDL, we build a sequential stage-wise learning architecture whose design philosophy embodies the model's predicted outcome and does not only depend on the current situation but also the history. Concretely, we meticulously combine the latest round of multimodal clinical data and the decayed past information to make assessments and predictions. Experimental results validate our assumption that sequential stage-wise learning outperforms single-stage learning, and that MMDL with multimodal inputs can beat any reduced model with single-modal inputs only.
Introduction
Clinical management of COVID-19 presents a significant triage challenge. While the majority of patients experience mild symptoms, a critical subset progresses rapidly to severe illness, necessitating intensive care and placing immense strain on medical infrastructure. Traditional AI models often suffer from data fragmentation—relying on a single snapshot or a single data type—which fails to capture the dynamic "big picture" of a patient's health trajectory.
The proposed Multistage Multimodal Deep Learning (MMDL) model addresses these gaps by treating the clinical journey as a sequence of multimodal data points. This allows the system to learn from the history of disease progression rather than just its instantaneous state, providing both current severity assessments and early warnings for potential deterioration.
Methods and Architecture
The MMDL model is built upon two primary innovations: a hierarchical feature extractor and a sequential stage-wise learning framework.
Two-Layer Multimodal Feature Extractor
To handle the diversity of clinical data, the model employs a two-layer extraction process:
- Intra-modal Hierarchy: Separate Multi-Layer Perceptron (MLP) networks extract latent features from each of the 13 modalities independently.
- Cross-modal Hierarchy: These independent features are concatenated and processed through another MLP to identify complex correlations between different data types, such as the relationship between inflammatory markers and renal function changes.
Sequential Stage-Wise Learning
Standard recurrent architectures like LSTMs often require long sequences to be effective. However, clinical data for COVID-19 patients is typically sparse, often consisting of only a few rounds of tests. MMDL utilizes a custom stage-wise architecture with a decay factor ($\alpha$). This mathematical approach allows the model to incorporate historical data while prioritizing the most recent clinical rounds, reflecting the reality that the most recent 48–72 hours are the most predictive of immediate outcomes.
Dataset and Patient Characteristics
The study utilized a clinical dataset of 216 COVID-19 patients from the Chongqing Public Health Medical Center. The cohort included 186 mild cases and 30 severe cases. The data encompassed 13 distinct modalities, including demographics, clinical manifestations, and 11 categories of laboratory tests (e.g., blood lipids, electrolytes, liver and renal function).
Clinical manifestations played a vital role in the initial assessment. Symptoms such as cough and fever were prevalent across the cohort, but their intensity and combination with other markers provided the necessary signals for the MMDL model to differentiate between mild and severe progression.
Key Results and Insights
The experiments confirmed that incorporating temporal history significantly enhances predictive power. Severity assessment accuracy improved from 96.26% (single-stage) to 98.10% when five consecutive rounds of data were used. For the high-risk "severe" group, the F1 score saw a substantial jump from 0.8091 to 0.9436.
A critical finding was the "Power of Multimodality." The full MMDL model consistently outperformed "reduced" versions that used only single modalities. Specifically, the fusion of inflammation markers, liver function, and blood lipids emerged as the most potent discriminator for disease severity.
While the model excelled at immediate and short-term forecasting, its performance in long-term early warning (predicting transitions up to four stages ahead) showed a decrease in F1 scores, highlighting the inherent complexity of long-term medical prognosis.
Conclusion
The MMDL model demonstrates that a holistic, temporal approach to clinical data can significantly improve the accuracy of COVID-19 severity assessments. By integrating diverse data modalities and accounting for the disease's progression over time, the model provides a robust tool for clinical decision-making. Future research will focus on scaling this framework to other chronic conditions that require continuous multimodal monitoring.
Summary updated