Preparing for the next pandemic via transfer learning from existing diseases with hierarchical multi-modal BERT: a study on COVID-19 outcome prediction

Khushbu Agarwal, Sutanay Choudhury, Sindhu Tipirneni, Pritam Mukherjee, Colby Ham, Suzanne Tamang, Matthew Baker, Siyi Tang, Veysel Kocaman, Olivier Gevaert, Robert Ralló, Chandan K. ReddyScientific 2022 · DOI

Abstract

Developing prediction models for emerging infectious diseases from relatively small numbers of cases is a critical need for improving pandemic preparedness. Using COVID-19 as an exemplar, we propose a transfer learning methodology for developing predictive models from multi-modal electronic healthcare records by leveraging information from more prevalent diseases with shared clinical characteristics. Our novel hierarchical, multi-modal model (TransMED) integrates baseline risk factors from the natural language processing of clinical notes at admission, time-series measurements of biomarkers obtained from laboratory tests, and discrete diagnostic, procedure and drug codes. We demonstrate the alignment of TransMED's predictions with well-established clinical knowledge about COVID-19 through univariate and multivariate risk factor driven sub-cohort analysis. TransMED's superior performance over state-of-the-art methods shows that leveraging patient data across modalities and transferring prior knowledge from similar disorders is critical for accurate prediction of patient outcomes, and this approach may serve as an important tool in the early response to future pandemics.

Introduction

When a new pandemic emerges, healthcare systems face a significant "cold start" problem. Rapid triage and resource allocation (e.g., ventilators) are essential, yet the historical data required to train accurate machine learning models is initially non-existent. Early COVID-19 models often relied on limited variables and simple regression, failing to capture the complex, multi-modal nature of electronic health records (EHR).

This study addresses two primary challenges: data scarcity and data heterogeneity. EHR data is a complex mixture of unstructured text, discrete codes (diagnoses, procedures, drugs), and continuous time-series data (vitals, labs). TransMED is designed to bridge this gap by "borrowing" knowledge from more common, similar conditions through transfer learning.

Methodology

TransMED is a hierarchical, multi-modal deep learning framework based on the BERT architecture. Its innovation lies in a two-step approach to learning:

  1. Patient Context Encoder: Instead of processing an entire hospital stay simultaneously, TransMED learns to represent a patient’s state within a single 24-hour "aggregation interval." It uses BERT layers to capture interactions between different data modalities (e.g., how a specific drug relates to a specific lab value) within that window.
  2. Temporal Modeling: These daily snapshots are then processed by a second layer (using GRU, Feed-Forward, or Attention mechanisms) to model the patient’s clinical evolution over time.
Overview of the hierarchical aggregation of patient context across time steps.

A key component is Transfer Learning. The model is pre-trained on a large cohort of patients with Severe Respiratory Diseases (SRD)—such as pneumonia, influenza, and ARDS—to learn the general "language" of respiratory failure. It is then fine-tuned on the smaller COVID-19 dataset.

Data and Multi-Modal Integration

The study utilized de-identified EHR data from Stanford Hospital, including a COVID-19 cohort (1,701 patients) and an SRD cohort (6,892 patients). The feature set is extensive, encompassing thousands of diagnosis, procedure, and drug codes, alongside binned laboratory measurements and clinical notes processed via NLP.

Example of the heterogeneous, multi-modal EHR data timeline for a single patient over 10 days.

The model integrates demographics, clinical notes, ICD-9/OMOP codes, and time-series biomarkers. The authors found a high medical vocabulary overlap (88.8% of diagnosis codes) between SRD and COVID-19, validating the transfer learning approach.

Results and Key Insights

TransMED significantly outperformed standard benchmarks, including Logistic Regression (LR) and standard BERT-based EHR models (BEHRT):

  • Ventilation Prediction: For 3-day ventilation risk, TransMED achieved an AUROC of 0.83, compared to 0.64 for LR and 0.62 for GRU.
  • Impact of Transfer Learning: Pre-training on the SRD cohort provided a 12.9% improvement in AUROC for predicting length of stay and a 10.3% improvement for ventilation prediction.
  • Hierarchical Advantage: Focusing on 24-hour windows allowed the model to handle the inherent sparsity and irregular intervals of medical data more effectively.
  • Multi-comorbidity Analysis: TransMED excelled at identifying complex feature interactions (e.g., CAD + Diabetes + Obesity) that simpler univariate models miss. Interestingly, procedure codes (like intubation) were found to be dominant predictors of patient severity.

Conclusion

TransMED demonstrates that leveraging multi-modal data and transferring knowledge from related disorders is critical for predicting outcomes in emerging diseases. While limitations exist regarding lab value consistency across datasets and demographic sparsity, the framework provides a robust foundation for early-response tools in future pandemics. Future work will explore extending this model to mortality prediction and automated drug recommendations.

Summary updated

Abstract

Developing prediction models for emerging infectious diseases from relatively small numbers of cases is a critical need for improving pandemic preparedness. Using COVID-19 as an exemplar, we propose a transfer learning methodology for developing predictive models from multi-modal electronic healthcare records by leveraging information from more prevalent diseases with shared clinical characteristics. Our novel hierarchical, multi-modal model ([Formula: see text]) integrates baseline risk factors from the natural language processing of clinical notes at admission, time-series measurements of biomarkers obtained from laboratory tests, and discrete diagnostic, procedure and drug codes. We demonstrate the alignment of [Formula: see text]'s predictions with well-established clinical knowledge about COVID-19 through univariate and multivariate risk factor driven sub-cohort analysis. [Formula: see text]'s superior performance over state-of-the-art methods shows that leveraging patient data across modalities and transferring prior knowledge from similar disorders is critical for accurate prediction of patient outcomes, and this approach may serve as an important tool in the early response to future pandemics.
BibTeX
@inproceedings{agarwal2022preparing,
  author = {Khushbu Agarwal and Sutanay Choudhury and Sindhu Tipirneni and Pritam Mukherjee and Colby Ham and Suzanne Tamang and Matthew Baker and Siyi Tang and Veysel Kocaman and Olivier Gevaert and Robert Ralló and Chandan K. Reddy},
  title = {{Preparing for the next pandemic via transfer learning from existing diseases with hierarchical multi-modal BERT: a study on COVID-19 outcome prediction}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41598-022-13072-w},
}
Figure