Identification of Adverse Drug Reaction Mentions in Tweets - SMM4H Shared Task 2019

Samarth Rawal, Siddharth Rawal, Saadat Anwar, Chitta Baral

Introduction

Pharmacovigilance—the science of monitoring drug effects after they reach the market—is critical for public safety. While clinical reports are the gold standard for identifying side effects, they are often slow to materialize. Social media, specifically Twitter, offers a real-time stream of patient experiences that can serve as an early warning system. However, finding the "needle in the haystack" is incredibly difficult: only about one in a thousand tweets is health-related, and even fewer contain actual Adverse Drug Reactions (ADRs).

Researchers from Arizona State University participated in the SMM4H (Social Media Mining for Health Applications) Shared Task 2019, developing a dual-pipeline system to address two specific challenges:

  1. Classification (Subtask 1): Determining if a tweet contains an ADR mention at all.
  2. Extraction (Subtask 2): Identifying the specific words or phrases that describe the reaction (e.g., "insomnia" or "skin rash").

Methodology

The team employed a "best tool for the job" approach, using different architectures for the two subtasks to balance performance with explainability.

Subtask 1: Lexicon-Based Classification

Instead of jumping straight to complex neural networks, the authors used a logistic regression model for classification. They extracted uni-grams and bi-grams from the training data and applied an "Internal + External Lexicon Selection" technique. They identified the top 700 features (keywords) based on absolute-valued coefficients and manually pruned irrelevant terms (like stop words). These curated terms were then used as binary features, allowing the model to remain interpretable while effectively filtering noisy social media text.

Subtask 2: Deep Learning for Named Entity Recognition (NER)

For the extraction task, the team implemented a BiLSTM-CRF (Bidirectional Long Short-Term Memory with a Conditional Random Field layer). This architecture is a standard but powerful choice for NER tasks, as it captures context from both directions of a sentence and models the dependencies between neighboring labels.

BiLSTM-CRF architecture for ADR extraction

To help the model understand medical context, they pre-initialized the embedding layer with weights from a word2vec model trained on the MIMIC-III dataset, which contains de-identified clinical data from hospital patients. This domain-specific initialization helped the model bridge the gap between informal "Twitter speak" and formal medical terminology.

Datasets

The primary dataset was provided by the SMM4H 2019 Shared Task, consisting of tweets labeled for ADR presence and specific ADR spans. Additionally, the MIMIC-III corpus was used as an external resource to ground the model's vocabulary in actual clinical language before it processed any social media data.

Key Insights and Results

The system's performance varied significantly based on the evaluation metrics, highlighting the challenges of unstructured text mining:

  • Classification Performance: The model achieved an F1-score of 0.4317, with a notably high Recall (0.6534) compared to Precision (0.3223). This suggests the model is effective at catching potential ADRs, making it a useful screening tool even if it occasionally flags non-ADR tweets.
  • Extraction Performance: Under the "relaxed" metric, the model achieved an F1-score of 0.535 (Recall: 0.753). However, under the "strict" metric, the F1-score dropped to 0.269. This gap indicates that while the model is excellent at finding the general vicinity of an ADR mention, getting the exact character-level boundaries correct in the chaotic syntax of a tweet remains a significant challenge.

One of the most important takeaways was the value of the "Human in the Loop." By manually paring down automatically generated keyword lists, the researchers demonstrated that expert intuition remains vital in refining NLP pipelines for specialized domains.

Future Work

While the BiLSTM-CRF approach proved robust, the authors noted that transformer-based models like BioBERT showed promise during preliminary training. Future iterations of this system will likely focus on integrating these state-of-the-art architectures and further tuning hyperparameters to close the performance gap between relaxed and strict extraction metrics.

Summary updated

Abstract

Analyzing social media posts can offer insights into a wide range of topics that are commonly discussed online, providing valuable information for studying various health-related phenomena reported online. The outcome of this work can offer insights into pharmacovigilance research to monitor the adverse effects of medications. This research specifically looks into mentions of adverse drug reactions (ADRs) in Twitter data through the Social Media Mining for Health Applications (SMM4H) Shared Task 2019. Adverse drug reactions are undesired harmful effects which can arise from medication or other methods of treatment. The goal of this research is to build accurate models using natural language processing techniques to detect reports of adverse drug reactions in Twitter data and extract these words or phrases.
BibTeX
@inproceedings{rawal2019identification,
  author = {Samarth Rawal and Siddharth Rawal and Saadat Anwar and Chitta Baral},
  title = {{Identification of Adverse Drug Reaction Mentions in Tweets - SMM4H Shared Task 2019}},
  booktitle = {SMM4H @ ACL},
  year = {2019},
  doi = {10.18653/v1/w19-3225},
}
Figure