MIDAS@SMM4H-2019: Identifying Adverse Drug Reactions and Personal Health Experience Mentions from Twitter

Debanjan Mahata, Sarthak Anand, Haimin Zhang, Simra Shahid, Laiba Mehnaz, Yaman Kumar, Rajiv Ratn ShahSMM4H @ ACL 2019 · DOI

Abstract

Public health surveillance can be significantly enhanced by mining informal patient reports on social media platforms like Twitter. This paper details Team MIDAS’s approach to the 2019 Social Media Mining for Health Applications (SMM4H) shared tasks, focusing on the identification of adverse drug reactions (ADRs) and personal health mentions. By leveraging state-of-the-art transfer learning models, specifically BERT and ULMFiT, we demonstrate that these architectures are remarkably robust to the extreme class imbalance inherent in social media data. Our results show that these models often outperform traditional sampling techniques when trained on original data distributions.

Introduction

While adverse drug reactions (ADRs) are a leading cause of death in the US, many incidents go unreported in clinical settings. Twitter offers a real-time window into patient experiences, but mining this data is a "needle in a haystack" problem.

The authors tackle three specific challenges:

  1. Classification (Task 1): Distinguishing tweets that mention an ADR from those that do not.
  2. Extraction (Task 2): Pinpointing the exact span of text describing the adverse effect.
  3. Personal Experience (Task 4): Identifying whether a user is reporting their own health status versus discussing a general health topic.

The primary technical hurdle is class imbalance. In the provided datasets, only about 9% of tweets contain ADR mentions. Traditionally, researchers use oversampling or undersampling to balance the data, but this paper investigates whether modern transfer learning can bypass that need.

Methods

The research employed a multi-pronged strategy using deep transfer learning:

  • ULMFiT (Universal Language Model Fine-tuning): A language model was fine-tuned on a large, domain-specific corpus of over 190,000 drug-related tweets before training the final classifier. This domain adaptation helps the model understand the specific linguistic nuances of health-related "Twitter-speak."
  • BERT: The standard BERT-base-uncased model was fine-tuned directly on the task-specific datasets.
  • Stacked Embeddings for Extraction: For the span extraction task (Task 2), a BiLSTM+CRF architecture was used. The model "stacked" different types of embeddings—combining the contextual power of BERT and Flair with the broad semantic knowledge of Twitter-specific GloVe embeddings.
  • Preprocessing: The ekphrasis library was used to handle social media noise, such as segmenting hashtags and normalizing tokens.

Datasets

The study used datasets provided by the SMM4H 2019 workshop:

  • Task 1: 25,672 tweets (2,374 ADR, 23,298 non-ADR).
  • Task 2: 2,367 tweets with labeled ADR spans.
  • Task 4: 10,876 tweets regarding personal health mentions (flu-related contexts).
  • External Data: 190,823 drug-related tweets used for pre-training the ULMFiT language model.

Results and Key Insights

One of the most significant findings was that BERT and ULMFiT performed best when trained on the original, imbalanced dataset. Attempts to undersample the data generally led to lower F1 scores, as illustrated in the figure below. This suggests that these high-capacity models are inherently better at handling skewed distributions than previous generations of classifiers.

F1 score validation across different undersampling ratios for BERT and ULMFiT.

Performance Summary

  • Task 1 (ADR Identification): ULMFiT achieved the highest F1 score of 0.5988, outperforming BERT and standard BiLSTM models.
  • Task 2 (Span Extraction): The BiLSTM+CRF model using a stack of BERT, Twitter GloVe, and Flair embeddings achieved a Relaxed F1 of 0.641.
  • Task 4 (Personal Health): A combination of BERT and Twitter GloVe embeddings yielded the best performance, with an overall F1 of 0.783.
Performance comparison of BERT, ULMFiT, and BLSTM on Task 1 (ADR Identification).

Limitations and Future Work

The authors noted several areas for improvement:

  • Computational Constraints: Due to high training costs and limited GPU resources, some BiLSTM models could not be fully optimized.
  • Model Interpretability: While the models perform well, improving interpretability is crucial for medical applications.
  • Specialized Embeddings: Future work will focus on training embeddings from scratch on even larger health-specific datasets to further improve the precision of ADR detection.

Summary updated

BibTeX
@inproceedings{mahata2019midas,
  author = {Debanjan Mahata and Sarthak Anand and Haimin Zhang and Simra Shahid and Laiba Mehnaz and Yaman Kumar and Rajiv Ratn Shah},
  title = {{MIDAS@SMM4H-2019: Identifying Adverse Drug Reactions and Personal Health Experience Mentions from Twitter}},
  booktitle = {SMM4H @ ACL},
  year = {2019},
  doi = {10.18653/v1/w19-3223},
}
Figure