Deep learning approaches for extracting adverse events and indications of dietary supplements from clinical text

Rui Zhang, PhD, Facmi, Famia, Fiahsi, Yadan Fan, Sicheng Zhou, Yifan Li, Rui Zhang

Abstract

Dietary supplements (DS) occupy a regulatory "gray area," often entering the market without the rigorous safety testing required for prescription drugs. Consequently, post-market surveillance is critical for identifying adverse events (AEs). This paper presents the first application of deep learning to extract DS-related safety signals directly from clinical notes in Electronic Health Records (EHRs). By employing BERT and attention-based LSTM models, the researchers successfully automated the identification of supplement mentions, their indications, and associated side effects, outperforming traditional machine learning baselines and uncovering potential health risks not yet documented in official databases.

Introduction

In the United States, approximately 77% of adults report taking dietary supplements. Despite their ubiquity, these products are regulated as food rather than drugs, meaning they do not undergo pre-market clinical trials to establish safety or efficacy. This regulatory gap places a heavy burden on post-market surveillance. Traditional methods, such as the FDA’s Adverse Event Reporting System (CAERS), rely on voluntary reporting, which is often incomplete, biased, and delayed.

Clinical notes within EHRs offer a rich, untapped source of real-world evidence. However, extracting meaningful data from these unstructured documents is a significant challenge. This study addresses this by developing a robust Natural Language Processing (NLP) framework to identify which supplements patients are using, why they are taking them (indications), and what adverse events they may be experiencing.

Methodology

The research team framed the problem as two distinct NLP tasks: Named Entity Recognition (NER) and Relation Extraction (RE).

Named Entity Recognition (NER)

The goal of NER was to identify mentions of supplements (e.g., garlic, folate) and clinical events (e.g., nausea, hot flashes). The researchers compared a traditional Conditional Random Field (CRF) baseline against several deep learning architectures:

  • Bi-LSTM-CRF: Utilizing word embeddings and character-level representations via CNNs or LSTMs.
  • BERT: Testing both the "Large Cased" version and "Clinical BERT" (pre-trained on ICU notes from the MIMIC database).

Relation Extraction (RE)

Once entities were identified, the model needed to determine the relationship between a supplement and an event. Relationships were classified into three categories: Positive (indication), Negative (adverse event), or Not Related. The models tested included:

  • Convolutional Neural Networks (CNNs): Utilizing position embeddings to capture local context.
  • Attention-based Bi-LSTM (Att-BLSTM): This model employs an attention mechanism to focus on the most contextually relevant words in a sentence to determine the relationship type.

Data and Validation

The study utilized data from the University of Minnesota’s Clinical Data Repository, encompassing 180 million notes from 2.9 million patients.

  1. NER Dataset: 7,000 sentences manually annotated for 7 common supplements.
  2. RE Dataset: 3,000 sentences annotated for 15 common supplements.
  3. Knowledge Discovery Set: To test real-world utility, the best-performing models were applied to a massive corpus of 26 million notes to search for 88 "unseen" supplements.

To verify the findings, results were compared against the Natural Medicines Comprehensive Database (NMCD), a gold standard for supplement information.

Results

Model Performance

The deep learning approaches significantly outperformed traditional baselines. For the NER task, the BERT Large model achieved the highest performance with an F1 score of 0.881, compared to the CRF baseline of 0.797. Interestingly, BERT Large outperformed Clinical BERT, suggesting that general-purpose pre-training may be more effective for the diverse clinical settings found in general EHR notes than models trained specifically on intensive care data.

For Relation Extraction, the Att-BLSTM model was the top performer, achieving an F1 score of 0.893. The attention mechanism proved vital in distinguishing between a patient taking a supplement for a condition versus a supplement causing a condition.

Real-World Discovery

When applied to the 88 unseen supplements, the model demonstrated high precision (e.g., 100% for Vitamin E and Biotin indications). Most importantly, the study found that approximately 29% of positive signals (indications) and 43% of negative signals (adverse events) identified by the model were not present in the NMCD. This highlights the potential for deep learning to discover novel safety signals that have not yet been officially documented.

Discussion and Limitations

The study demonstrates that deep learning can effectively navigate the complexities of clinical text to monitor supplement safety. Neural models showed particular strength in handling imbalanced data, where "Events" appeared far less frequently than supplement names—a common hurdle for traditional machine learning.

However, several limitations remain. The manual annotation of clinical text is resource-intensive, resulting in a relatively small training set. Furthermore, the current models are limited to intra-sentence relations. If a supplement and its effect are mentioned in separate sentences, the model cannot currently link them. Future work will focus on expanding the dataset and developing cross-sentence relation extraction to create a more comprehensive pharmacovigilance system.

Conclusion

By leveraging BERT and attention-based architectures, this research provides a scalable path forward for monitoring the "Wild West" of dietary supplements. Moving beyond voluntary reporting to automated EHR mining allows for a more proactive approach to public health, ensuring that safety signals are identified and investigated in real-time.

Summary updated

Abstract

The objective of our work is to demonstrate the feasibility of utilizing deep learning models to extract safety signals related to the use of dietary supplements (DS) in clinical text. Two tasks were performed in this study. For the named entity recognition (NER) task, Bi-LSTM-CRF (Bidirectional Long-Short-Term-Memory Conditional Random Fields) and BERT (Bidirectional Encoder Representations from Transformers) models were trained and compared with CRF model as a baseline to recognize the named entities of DS and Events from clinical notes. In the relation extraction (RE) task, two deep learning models, including attention-based Bi-LSTM and CNN (Convolutional Neural Network), and a random forest model were trained to extract the relations between DS and Events, which were categorized into three classes: positive (i.e., indication), negative (i.e., adverse events), and not related. The best performed NER and RE models were further applied on clinical notes mentioning 88 DS for discovering DS adverse events and indications, which were compared with a DS knowledge base. For the NER task, deep learning models achieved a better performance than CRF, with F1 scores above 0.860. The attention-based Bi-LSTM model performed the best in the relation extraction task, with the F1 score of 0.893. When comparing DS event pairs generated by the deep learning models with the knowledge base for DS and Event, we found both known and unknown pairs. Deep learning models can detect adverse events and indication of DS in clinical notes, which hold great potential for monitoring the safety of DS use.
BibTeX
@article{fiahsi2021deep,
  author = {Rui Zhang, PhD, FACMI, FAMIA, FIAHSI and Yadan Fan and Sicheng Zhou and Yifan Li and Rui Zhang},
  title = {{Deep learning approaches for extracting adverse events and indications of dietary supplements from clinical text}},
  journal = {J. Am. Medical Informatics Assoc},
  year = {2021},
  doi = {10.1093/jamia/ocaa218},
}
Figure