Abstract
Pretraining electronic health record (EHR) data using language models by treating patient trajectories as natural language sentences has enhanced performance across various medical tasks. However, EHR pretraining models have never been utilized in adverse drug event (ADE) prediction. A retrospective study was conducted on observational medical outcomes partnership (OMOP)-common data model (CDM) based EHR data from two separate tertiary hospitals. For pretraining, codes were randomly masked, and the model was trained to infer the masked tokens utilizing preceding and following history. In this process, we introduced domain embedding (DE) to provide information about the domain of the masked token, preventing the model from finding codes from irrelevant domains. EHR pretraining model with DE outperformed all the other baselines in all cohorts. For feature importance analysis, we demonstrated that the results were consistent with priorly reported background clinical knowledge. CDM-based EHR pretraining model with DE is a proper model for various ADE prediction tasks.
Introduction
Adverse Drug Events (ADEs)—injuries resulting from medical interventions related to a drug—are a leading cause of hospital morbidity and mortality. Traditional methods for detecting ADEs often rely on retrospective statistical analysis or simple machine learning models that struggle to capture the complex, temporal "story" of a patient’s health. This paper presents a novel application of transformer-based language models to the critical field of pharmacovigilance. By treating a patient’s medical history as a "sentence" and individual medical codes as "words," the researchers developed a system to predict ADEs with high accuracy across different hospital systems.
Methods: Domain-Aware Pretraining
The researchers adapted the BERT architecture to pretrain on patient trajectories from Electronic Health Records (EHR). The model (6 layers, 8 attention heads) was trained using Masked Language Modeling (MLM), where 30% of the medical codes in a patient’s history are hidden, and the model must predict them based on the surrounding context.
As illustrated in Figure 1, the key innovation is the introduction of Domain Embedding (DE). Medical codes are drawn from different "domains" (e.g., diagnosis, drug, measurement, or procedure). By adding a specific embedding that identifies the domain of a masked token, the researchers provided a "hint" that narrowed the search space. This prevented the model from attempting to predict a drug code when the masked slot was clearly a lab measurement, significantly improving learning efficiency.
Experimental Setup and Validation
The study utilized massive standardized datasets from two major tertiary hospitals in South Korea: Seoul National University Hospital (SNUH) and Ajou University Medical Center (AUMC). The researchers focused on three specific ADE cohorts: NSAID-induced Peptic Ulcers (NSAID-PU), anticoagulant-induced Intracranial Hemorrhage (AC-ICH), and chemotherapy-induced Neutropenic Fever (Chemo-NF).
To evaluate the model's robustness, the team implemented a rigorous validation framework (Figure 3). This included internal validation at Hospital A and external validation at Hospital B using three transfer strategies. The "Type II" approach—taking a model pretrained on Hospital A, performing a short "warm-up" pretraining on Hospital B, and then fine-tuning—yielded the best results, successfully mitigating the data distribution gap between institutions.
Key Results and Insights
The pretrained model with Domain Embedding consistently outperformed all baselines, including models initialized with randomized weights and pretrained models without DE.
- Internal Validation (AUROC): The model achieved exceptional performance with scores of 0.977 for NSAID-PU, 0.908 for AC-ICH, and 0.989 for Chemo-NF.
- Clinical Interpretability: Analysis of the model’s attention matrices revealed that it focused on clinically relevant features. For example, in the anticoagulant cohort, the model paid high attention to "skin lesions," a logical clinical precursor to internal bleeding.
- Transferability: Despite low overlap in specific medical codes between hospitals (only ~7,400 shared out of 20,000+), the pretraining on standardized CDM data allowed for effective knowledge transfer.
Conclusion
CDM-based EHR pretraining with Domain Embedding is a powerful tool for ADE prediction. By providing categorical context during the learning phase, the model more effectively captures the underlying logic of medical trajectories. Future work aims to incorporate unstructured data, such as clinical notes, and integrate Large Language Models to provide natural-language explanations for high-risk flags, enhancing utility for bedside clinicians.
Summary updated