Abstract
Dietary supplements (DS) occupy a regulatory "gray area," often entering the market without the rigorous safety testing required for prescription drugs. Consequently, post-market surveillance is critical for identifying adverse events (AEs). This paper presents the first application of deep learning to extract DS-related safety signals directly from clinical notes in Electronic Health Records (EHRs). By employing BERT and attention-based LSTM models, the researchers successfully automated the identification of supplement mentions, their indications, and associated side effects, outperforming traditional machine learning baselines and uncovering potential health risks not yet documented in official databases.
Introduction
In the United States, approximately 77% of adults report taking dietary supplements. Despite their ubiquity, these products are regulated as food rather than drugs, meaning they do not undergo pre-market clinical trials to establish safety or efficacy. This regulatory gap places a heavy burden on post-market surveillance. Traditional methods, such as the FDA’s Adverse Event Reporting System (CAERS), rely on voluntary reporting, which is often incomplete, biased, and delayed.
Clinical notes within EHRs offer a rich, untapped source of real-world evidence. However, extracting meaningful data from these unstructured documents is a significant challenge. This study addresses this by developing a robust Natural Language Processing (NLP) framework to identify which supplements patients are using, why they are taking them (indications), and what adverse events they may be experiencing.
Methodology
The research team framed the problem as two distinct NLP tasks: Named Entity Recognition (NER) and Relation Extraction (RE).
Named Entity Recognition (NER)
The goal of NER was to identify mentions of supplements (e.g., garlic, folate) and clinical events (e.g., nausea, hot flashes). The researchers compared a traditional Conditional Random Field (CRF) baseline against several deep learning architectures:
- Bi-LSTM-CRF: Utilizing word embeddings and character-level representations via CNNs or LSTMs.
- BERT: Testing both the "Large Cased" version and "Clinical BERT" (pre-trained on ICU notes from the MIMIC database).
Relation Extraction (RE)
Once entities were identified, the model needed to determine the relationship between a supplement and an event. Relationships were classified into three categories: Positive (indication), Negative (adverse event), or Not Related. The models tested included:
- Convolutional Neural Networks (CNNs): Utilizing position embeddings to capture local context.
- Attention-based Bi-LSTM (Att-BLSTM): This model employs an attention mechanism to focus on the most contextually relevant words in a sentence to determine the relationship type.
Data and Validation
The study utilized data from the University of Minnesota’s Clinical Data Repository, encompassing 180 million notes from 2.9 million patients.
- NER Dataset: 7,000 sentences manually annotated for 7 common supplements.
- RE Dataset: 3,000 sentences annotated for 15 common supplements.
- Knowledge Discovery Set: To test real-world utility, the best-performing models were applied to a massive corpus of 26 million notes to search for 88 "unseen" supplements.
To verify the findings, results were compared against the Natural Medicines Comprehensive Database (NMCD), a gold standard for supplement information.
Results
Model Performance
The deep learning approaches significantly outperformed traditional baselines. For the NER task, the BERT Large model achieved the highest performance with an F1 score of 0.881, compared to the CRF baseline of 0.797. Interestingly, BERT Large outperformed Clinical BERT, suggesting that general-purpose pre-training may be more effective for the diverse clinical settings found in general EHR notes than models trained specifically on intensive care data.
For Relation Extraction, the Att-BLSTM model was the top performer, achieving an F1 score of 0.893. The attention mechanism proved vital in distinguishing between a patient taking a supplement for a condition versus a supplement causing a condition.
Real-World Discovery
When applied to the 88 unseen supplements, the model demonstrated high precision (e.g., 100% for Vitamin E and Biotin indications). Most importantly, the study found that approximately 29% of positive signals (indications) and 43% of negative signals (adverse events) identified by the model were not present in the NMCD. This highlights the potential for deep learning to discover novel safety signals that have not yet been officially documented.
Discussion and Limitations
The study demonstrates that deep learning can effectively navigate the complexities of clinical text to monitor supplement safety. Neural models showed particular strength in handling imbalanced data, where "Events" appeared far less frequently than supplement names—a common hurdle for traditional machine learning.
However, several limitations remain. The manual annotation of clinical text is resource-intensive, resulting in a relatively small training set. Furthermore, the current models are limited to intra-sentence relations. If a supplement and its effect are mentioned in separate sentences, the model cannot currently link them. Future work will focus on expanding the dataset and developing cross-sentence relation extraction to create a more comprehensive pharmacovigilance system.
Conclusion
By leveraging BERT and attention-based architectures, this research provides a scalable path forward for monitoring the "Wild West" of dietary supplements. Moving beyond voluntary reporting to automated EHR mining allows for a more proactive approach to public health, ensuring that safety signals are identified and investigated in real-time.
Summary updated