Preparing For The Next Pandemic: Transfer Learning From Existing Diseases Via Hierarchical Multi-Modal BERT Models to Predict COVID-19 Outcomes

Khushbu Agarwal, Sutanay Choudhury, Sindhu Tipirneni, Pritam Mukherjee, Colby Ham, Suzanne Tamang, Matthew Baker, Siyi Tang, Veysel Kocaman, Olivier Gevaert, Robert Ralló, Chandan K. ReddyResearch 2021 · DOI

Introduction

When a novel pandemic emerges, healthcare systems face a critical "cold start" problem. Predictive models, essential for resource triaging and risk assessment, typically require vast amounts of data. However, in the early stages of an outbreak, such data is scarce. Furthermore, Electronic Health Records (EHR) are complex and heterogeneous, comprising unstructured clinical notes, discrete diagnostic codes, medication orders, and irregular time-series laboratory measurements.

To address these challenges, we present TRANSMED, a hierarchical, multi-modal deep learning framework. TRANSMED predicts patient outcomes for emerging diseases by leveraging knowledge from more common, clinically similar illnesses through transfer learning. By pre-training on a large cohort of Severe Respiratory Disease (SRD) patients, the model achieves significant performance gains in predicting ventilation risk and hospital stay duration for COVID-19, even when specific disease data is limited.

Methodology: The TRANSMED Architecture

TRANSMED is designed to handle the inherent complexity and multi-modality of EHR data through a two-level hierarchical architecture.

Figure 1: The TRANSMED framework integrates multi-modal data (notes, codes, labs) into daily snapshots, which are then processed temporally to predict patient trajectories.
  1. Multi-Modal Patient Context Encoders: At the primary level, the model utilizes BERT-based layers to learn interactions between disparate data types—including clinical notes (processed via NLP), ICD codes, and laboratory results—within a single 24-hour aggregation interval. This allows the model to "contextualize" data; for instance, recognizing the increased significance of a specific medication when paired with a particular lab result.
  2. Temporal Modeling Layer: The secondary level processes these daily "snapshots" over time using a Gated Recurrent Unit (GRU) or a temporal BERT layer to predict the patient's clinical trajectory.
  3. Transfer Learning Strategy: The model is first trained on a large dataset of patients with Severe Respiratory Disease (SRD), such as influenza and pneumonia. These conditions share significant clinical characteristics with COVID-19. The knowledge gained from this "source" domain is then fine-tuned on the smaller COVID-19 "target" dataset.
  4. Self-Supervised Pre-training: Utilizing a "masked language modeling" approach, the model learns the underlying structure of medical data by predicting missing codes, enabling effective learning without requiring explicit labels for every data point.

Datasets and Cohort Selection

The study utilized de-identified EHR data from the Stanford Hospital (STARR repository) spanning 2015 to 2021.

Figure 2: Flowcharts illustrating the inclusion and exclusion criteria for the COVID-19 and Severe Respiratory Disease (SRD) cohorts.
  • SRD Cohort (Pre-training): Consisted of 6,892 patients (9,348 hospitalizations) diagnosed with pneumonia, influenza, or acute respiratory distress syndrome.
  • COVID-19 Cohort (Fine-tuning): Included 1,701 adult patients.
  • Data Modalities: The model integrated free-text admission notes, ICD-9/10 codes, procedure codes, drug codes, and eight high-frequency laboratory measurements (e.g., lymphocytes, eosinophils, and alanine aminotransferase).

Analysis revealed a massive overlap in medical "vocabulary" between the SRD and COVID-19 cohorts (e.g., 88.8% of diagnosis codes and 94% of drug codes), validating the use of SRD as a source for transfer learning.

Key Results and Clinical Alignment

TRANSMED was evaluated on two primary tasks: predicting the likelihood of a long hospital stay and the risk of requiring mechanical ventilation (with 3-day and 7-day look-ahead windows).

  • Performance Gains: TRANSMED consistently outperformed standard baselines, including Logistic Regression, GRUs, and BEHRT. For the 7-day ventilation risk task, TRANSMED achieved an AUROC of 0.77, a 17.5% average improvement over the best non-transfer-learning baseline.
  • Impact of Transfer Learning: Ablation studies demonstrated that transfer learning from the SRD cohort provided a performance boost of 8–17% across different tasks.
  • Robustness: The model showed significant improvements in F1 scores (up to 66% better than baselines), indicating superior performance in handling imbalanced data where positive outcomes (like ventilation) are rare.
  • Clinical Validity: The model's risk scores align with established clinical knowledge. For example, it correctly identified that patients with comorbid Coronary Artery Disease (CAD) and Diabetes were at the highest risk for ventilation.

Discussion and Future Directions

While TRANSMED shows great promise, some limitations remain. Data sparsity in laboratory results necessitated limiting inputs to only eight high-frequency tests. Additionally, demographic shifts between the SRD and COVID-19 cohorts (such as age and ethnicity distributions) require careful management to avoid bias.

Future work will focus on extending this framework to "multi-way comorbidity analysis," capturing even more complex interactions between multiple pre-existing conditions to further refine patient risk stratification during the early stages of future pandemics.

Summary updated

Abstract

Abstract Developing prediction models for emerging infectious diseases from relatively small numbers of cases is a critical need for improving pandemic preparedness. Using COVID-19 as an exemplar, we propose a transfer learning methodology for developing predictive models from multi-modal electronic healthcare records by leveraging information from more prevalent diseases with shared clinical characteristics. Our novel hierarchical, multi-modal model (TransMED) integrates baseline risk factors from the natural language processing of clinical notes at admission, time-series measurements of biomarkers obtained from laboratory tests, and discrete diagnostic, procedure and drug codes. We demonstrate the alignment of TransMED's predictions with well-established clinical knowledge about COVID-19 through univariate and multivariate risk factor driven sub-cohort analysis. TransMED's superior performance over state-of-the-art methods shows that leveraging patient data across modalities and transferring prior knowledge from similar disorders is critical for accurate prediction of patient outcomes, and this approach may serve as an important tool in the early response to future pandemics.
BibTeX
@inproceedings{agarwal2021preparing,
  author = {Khushbu Agarwal and Sutanay Choudhury and Sindhu Tipirneni and Pritam Mukherjee and Colby Ham and Suzanne Tamang and Matthew Baker and Siyi Tang and Veysel Kocaman and Olivier Gevaert and Robert Ralló and Chandan K. Reddy},
  title = {{Preparing For The Next Pandemic: Transfer Learning From Existing Diseases Via Hierarchical Multi-Modal BERT Models to Predict COVID-19 Outcomes}},
  booktitle = {Research},
  year = {2021},
  doi = {10.21203/rs.3.rs-965815/v1},
}
Figure