SUPP.AI: finding evidence for supplement-drug interactions

Lucy Lu Wang, Oyvind Tafjord, Arman Cohan, Sarthak Jain, Sam Skjonsberg, Carissa Schoenick, Nick Botner, Waleed Ammar

Abstract

Researchers from the Allen Institute for AI (AI2) have developed SUPP.AI, a specialized search engine and NLP pipeline that automatically extracts evidence of supplement-drug interactions (SDIs) from over 22 million biomedical articles. By leveraging transfer learning from drug-drug interaction (DDI) data, the system surfaced 195,000 evidence sentences for 60,000 unique interactions, providing a critical safety resource for clinicians and consumers.

Introduction

While more than half of US adults take dietary supplements—including vitamins, minerals, and herbs—our understanding of how these substances interact with pharmaceutical drugs remains dangerously incomplete. These interactions can lead to significant adverse effects, yet the relevant information is often buried within massive volumes of scientific literature.

Traditional manual curation by experts is slow, expensive, and unable to keep pace with the thousands of new papers published monthly. Furthermore, existing clinical databases often require expensive subscriptions, leaving many healthcare providers and consumers without accessible information. SUPP.AI aims to bridge this "information gap" by automating the extraction of interaction evidence using state-of-the-art machine learning.

Methods and Datasets

The authors developed a multi-stage NLP pipeline to identify and classify interactions across a massive corpus of 22 million Medline-indexed articles from Semantic Scholar.

The NLP Pipeline

  1. Entity Linking: The system uses scispaCy and the Unified Medical Language System (UMLS) to identify mentions of supplements and drugs. This process involves mapping text spans to standardized medical concepts.
  2. Transfer Learning: Due to the scarcity of labeled data specifically for supplement interactions, the team fine-tuned a RoBERTa language model on existing datasets for Drug-Drug Interactions (DDIs), such as DDI-2013 and NLM-DailyMed. They hypothesized that the linguistic patterns describing drug-drug interactions are sufficiently similar to those describing supplement-drug interactions.
  3. Generalization via Masking: To prevent the model from memorizing specific drug pairs, entity names were replaced with generic tokens (e.g., [Arg1] and [Arg2]). This forces the model to focus on the contextual language of the interaction.
  4. Evidence Ranking: Extracted sentences are surfaced in a web application and ranked based on recency and the "level of evidence" (e.g., prioritizing clinical trials over animal studies).
Entity linking of supplement (garlic) and drug (warfarin) mentions within the pipeline.

Evaluation Datasets

To evaluate the model, the authors created Supplements-500, a new, manually annotated test set of 500 sentences involving supplement interactions. The final SUPP.AI Dataset contains 195k evidence sentences supporting 60k unique interactions.

Key Results and Insights

The research demonstrated that knowledge of pharmaceutical drug interactions transfers effectively to the supplement domain. The RoBERTa-DDI model achieved a state-of-the-art F1-score of 0.89 on the DDI-2013 detection task. When applied to supplements, it maintained a high Precision of 0.82, ensuring that the results shown to users are reliable, despite a lower recall (0.58).

The SUPP.AI interface displaying identified interactions and supporting research papers for Ginkgo Biloba.

The system successfully mapped 60,000 interactions involving 2,044 unique supplements and 2,772 unique drugs, sourced from 133,000 different papers. A key insight from this work is that the primary bottleneck is not the classification model itself, but rather Named Entity Recognition (NER). Automated tools correctly recognized only about 80% of drug entities, and supplement terminology is even less standardized, creating a performance ceiling.

Limitations and Future Work

Current medical ontologies have poor coverage of supplement terminology (estimated between 14-54%), making many niche herbal products "invisible" to the system. Additionally, the reliance on automated entity linkers leads to missed interactions. Future work aims to move beyond simple detection to classify the type of interaction (e.g., "increases toxicity" vs. "decreases efficacy") and provide more nuanced evidence scoring.

Summary updated

BibTeX
@inproceedings{wang2020supp,
  author = {Lucy Lu Wang and Oyvind Tafjord and Arman Cohan and Sarthak Jain and Sam Skjonsberg and Carissa Schoenick and Nick Botner and Waleed Ammar},
  title = {{SUPP.AI: finding evidence for supplement-drug interactions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-demos.41},
}
Figure