Abstract
Developing prediction models for emerging infectious diseases from relatively small numbers of cases is a critical need for improving pandemic preparedness. Using COVID-19 as an exemplar, we propose a transfer learning methodology for developing predictive models from multi-modal electronic healthcare records by leveraging information from more prevalent diseases with shared clinical characteristics. Our novel hierarchical, multi-modal model (TransMED) integrates baseline risk factors from the natural language processing of clinical notes at admission, time-series measurements of biomarkers obtained from laboratory tests, and discrete diagnostic, procedure and drug codes. We demonstrate the alignment of TransMED's predictions with well-established clinical knowledge about COVID-19 through univariate and multivariate risk factor driven sub-cohort analysis. TransMED's superior performance over state-of-the-art methods shows that leveraging patient data across modalities and transferring prior knowledge from similar disorders is critical for accurate prediction of patient outcomes, and this approach may serve as an important tool in the early response to future pandemics.
Introduction
When a new pandemic emerges, healthcare systems face a significant "cold start" problem. Rapid triage and resource allocation (e.g., ventilators) are essential, yet the historical data required to train accurate machine learning models is initially non-existent. Early COVID-19 models often relied on limited variables and simple regression, failing to capture the complex, multi-modal nature of electronic health records (EHR).
This study addresses two primary challenges: data scarcity and data heterogeneity. EHR data is a complex mixture of unstructured text, discrete codes (diagnoses, procedures, drugs), and continuous time-series data (vitals, labs). TransMED is designed to bridge this gap by "borrowing" knowledge from more common, similar conditions through transfer learning.
Methodology
TransMED is a hierarchical, multi-modal deep learning framework based on the BERT architecture. Its innovation lies in a two-step approach to learning:
- Patient Context Encoder: Instead of processing an entire hospital stay simultaneously, TransMED learns to represent a patient’s state within a single 24-hour "aggregation interval." It uses BERT layers to capture interactions between different data modalities (e.g., how a specific drug relates to a specific lab value) within that window.
- Temporal Modeling: These daily snapshots are then processed by a second layer (using GRU, Feed-Forward, or Attention mechanisms) to model the patient’s clinical evolution over time.
A key component is Transfer Learning. The model is pre-trained on a large cohort of patients with Severe Respiratory Diseases (SRD)—such as pneumonia, influenza, and ARDS—to learn the general "language" of respiratory failure. It is then fine-tuned on the smaller COVID-19 dataset.
Data and Multi-Modal Integration
The study utilized de-identified EHR data from Stanford Hospital, including a COVID-19 cohort (1,701 patients) and an SRD cohort (6,892 patients). The feature set is extensive, encompassing thousands of diagnosis, procedure, and drug codes, alongside binned laboratory measurements and clinical notes processed via NLP.
The model integrates demographics, clinical notes, ICD-9/OMOP codes, and time-series biomarkers. The authors found a high medical vocabulary overlap (88.8% of diagnosis codes) between SRD and COVID-19, validating the transfer learning approach.
Results and Key Insights
TransMED significantly outperformed standard benchmarks, including Logistic Regression (LR) and standard BERT-based EHR models (BEHRT):
- Ventilation Prediction: For 3-day ventilation risk, TransMED achieved an AUROC of 0.83, compared to 0.64 for LR and 0.62 for GRU.
- Impact of Transfer Learning: Pre-training on the SRD cohort provided a 12.9% improvement in AUROC for predicting length of stay and a 10.3% improvement for ventilation prediction.
- Hierarchical Advantage: Focusing on 24-hour windows allowed the model to handle the inherent sparsity and irregular intervals of medical data more effectively.
- Multi-comorbidity Analysis: TransMED excelled at identifying complex feature interactions (e.g., CAD + Diabetes + Obesity) that simpler univariate models miss. Interestingly, procedure codes (like intubation) were found to be dominant predictors of patient severity.
Conclusion
TransMED demonstrates that leveraging multi-modal data and transferring knowledge from related disorders is critical for predicting outcomes in emerging diseases. While limitations exist regarding lab value consistency across datasets and demographic sparsity, the framework provides a robust foundation for early-response tools in future pandemics. Future work will explore extending this model to mortality prediction and automated drug recommendations.
Summary updated