Development and Validation of a Multimodal-Based Prognosis and Intervention Prediction Model for COVID-19 Patients in a Multicenter Cohort

Jeonghoon Lee, Jong Seok Ahn, Myung Jin Chung, Yeon Joo Jeong, Jin Hwan Kim, Jae Kwang Lim, Young Jae Kim, Jong Eun Lee, Eun Young Kim2022 · DOI

TL;DR

This paper introduces a multimodal AI model that combines chest X-ray (CXR) features, clinical data, and lab results to predict the prognosis and intervention needs of COVID-19 patients. The multimodal approach significantly outperforms single-modality models, offering a more comprehensive and accurate prediction tool for clinical decision-making and resource management.

What Problem Does This Paper Address?

The COVID-19 pandemic placed immense strain on healthcare systems, necessitating efficient patient triaging, prognostication, and resource allocation. Accurately predicting which patients would require intensive care, mechanical ventilation, or face in-hospital mortality is crucial for effective management, especially in resource-limited settings. While AI has shown promise in analyzing medical images like CXRs, and clinical data is vital, a unified approach that leverages all these data sources to predict both prognosis and intervention needs has been lacking. Existing studies often focused on single data types or only prognosis, not the critical aspect of predicting required interventions. This paper aims to bridge this gap by developing and validating a multimodal AI model for comprehensive COVID-19 patient management.

Methods

The study employs a two-step multimodal approach. First, an established, commercially available AI engine, DLAD-10 (Lunit INSIGHT for Chest Radiography), was used to extract features from initial chest X-rays. This engine is trained to detect 10 common CXR abnormalities and provides outputs (values between 0 and 100) representing the probability of these abnormalities. These AI-interpreted CXR features were then combined with clinical data (demographics, comorbidities, symptoms) and laboratory data (lymphocyte count, platelet count, CRP, LDH).

For prediction, a random forest model with a quantile classifier was utilized. This ensemble method was trained to predict various outcomes, including oxygen supplementation, mechanical ventilation, ECMO, ICU admission, and in-hospital mortality, as well as a composite "all adverse events" outcome. Feature selection was performed using mean decrease in accuracy and mean decrease in Gini to identify the most important predictors.

Datasets

The study utilized a multicenter dataset of 2282 RT-PCR-confirmed COVID-19 patients from 13 referral institutions across South Korea. Data was collected retrospectively between January 2020 and June 2021. The dataset included initial clinical findings, laboratory data, and chest X-rays. The data was partitioned into a development set (1731 patients from 8 institutions) and a validation set (551 patients from 5 institutions) to ensure robust external validation.

Key Insights and Ideas

  • Multimodality is Key: Combining diverse data sources (CXR, clinical, lab) is significantly more effective than relying on any single source for predicting COVID-19 outcomes and intervention needs. This mirrors how clinicians integrate information.
  • Explainable AI for Clinical Trust: Using an AI engine that outputs clinically interpretable CXR features (e.g., "consolidation") rather than abstract numerical outputs allows clinicians to understand why a prediction is made, fostering trust and adoption.
  • Intervention Prediction is Crucial: Predicting not just mortality but also the need for interventions like oxygen or mechanical ventilation is vital for resource management, especially in surge situations.
  • Data Partitioning Matters: Splitting data by institution for validation, rather than just random patient-level splitting, better reflects real-world variability in healthcare settings and ensures more generalizable model performance.

Key Results

The multimodal model achieved a significantly higher Area Under the Receiver Operating Characteristic (AUROC) of 0.854 (95% CI, 0.820–0.889) for predicting all adverse events, compared to single-modal models: clinical findings (AUROC 0.742), laboratory data (AUROC 0.794), and CXR-based (AUROC 0.770). This improvement was statistically significant (p < 0.001).

For individual adverse events, the multimodal model generally outperformed single-modal models, with the exception of ECMO prediction, which had a very small number of cases. The most important predictors for all adverse events were identified as CRP (laboratory data), age (clinical data), dyspnea (clinical data), and consolidation (CXR feature).

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Imbalance: The training and validation datasets were not perfectly balanced in terms of predicted outcomes, reflecting real-world hospital resource availability.
  • Single CXR Use: Only the initial CXR at admission was used. Future work could incorporate serial CXRs to capture disease progression.
  • No Temporal Analysis: The study did not consider the sequential or temporal relationship between clinical data and outcomes. Future research could focus on models predicting the timeframe for events or interventions.

The authors suggest that the ensemble approach and the use of clinically interpretable features could be applied to other prediction models and emerging diseases.

Summary updated

BibTeX
@misc{lee2022development,
  author = {Jeonghoon Lee and Jong Seok Ahn and Myung Jin Chung and Yeon Joo Jeong and Jin Hwan Kim and Jae Kwang Lim and Jin Young Kim and Young Jae Kim and Jong Eun Lee and Eun Young Kim},
  title = {{Development and Validation of a Multimodal-Based Prognosis and Intervention Prediction Model for COVID-19 Patients in a Multicenter Cohort}},
  year = {2022},
  doi = {10.3390/s22135007},
}
Figure