Towards Text Processing Pipelines to Identify Adverse Drug Events-related Tweets: University of Michigan @ SMM4H 2019 Task 1

V.G.Vinod Vydiswaran, Grace Ganzel, Bryan Romas, Deahan Yu, Amy Austin, Neha Bhomia, Socheatha Chan, Stephanie Hall, Van Le, Aaron Miller, Olawunmi Oduyebo, Aulia Song, Radhika Sondhi, Danny Teng, Hao Tseng, Kim Vuong, Stephanie ZimmermanACL 2019

TL;DR

This paper presents a robust text processing pipeline for identifying adverse drug event (ADE) mentions in social media. Their approach combines traditional machine learning and deep learning models, achieving above-average performance in the SMM4H 2019 Shared Task. This work highlights the potential of social media data for public health surveillance and pharmacovigilance.

What Problem Does This Paper Address?

The increasing volume of health-related discussions on social media presents a valuable, yet challenging, resource for public health monitoring and pharmacovigilance. Specifically, identifying mentions of adverse drug events (ADEs) in tweets is crucial for understanding drug safety and potential public health concerns. However, tweets are short, informal, and often contain subtle linguistic variations that make distinguishing ADEs from indications for drug use a difficult task. The SMM4H 2019 Shared Task aimed to address this challenge by providing a benchmark for systems to accurately classify tweets reporting ADEs.

Methods

The authors developed a multi-stage text processing pipeline to prepare tweets for analysis. Key steps included:

  • UTF-8 Character Handling: Emojis and other non-standard characters were replaced with descriptive tags (e.g., 'pill' emoji to '(pill)').
  • Ekphrasis Tool: This tool was used for text normalization, with modifications to handle uppercase contractions and expand them. Tweets were then lowercased for full processing.
  • NLTK TweetTokenizer and Lemmatizer: Standard NLP tools were employed to tokenize tweets and reduce words to their base or dictionary form.
  • MetaMap: This tool was used to identify medical concepts and semantic types within the text, with concepts scoring above 800 being extracted as features.
  • cTAKES: This system was used to identify concepts from the Systematized Nomenclature of Medicine (SNOMED), with identified codes serving as features.
  • Pattern-based Features: Regular expressions were used to generate additional features based on specific patterns.

For modeling, they explored two main neural network architectures: a bidirectional LSTM and a bidirectional LSTM with a CNN layer. These models were trained using pre-trained GloVe and Word2vec embeddings. They also experimented with traditional machine learning models, including a linear kernel Support Vector Machine (SVM).

Datasets

The primary dataset used was the SMM4H 2019 Task 1 dataset, which provided labeled tweets for ADE classification. The paper mentions using a MedDRA dictionary from SIDER for medical dictionary features.

Key Insights and Ideas

  • The importance of robust pre-processing: The paper emphasizes that effective handling of informal text, including emojis and contractions, is critical for downstream ADE identification.
  • Feature engineering pays off: Combining various features, from medical dictionaries and concept extraction (MetaMap, cTAKES) to part-of-speech tags and character embeddings, significantly improved performance.
  • Deep learning models show promise: While traditional models performed well, the bidirectional LSTM with GloVe embeddings emerged as the best performing model, suggesting the power of contextual word representations.
  • Subtle linguistic cues matter: The challenge of distinguishing ADEs from indications highlights the need for models that can capture nuanced semantic differences in short texts.

Key Results

The authors submitted three runs to the SMM4H 2019 Task 1. Their best performing run (Run 2) achieved an F1 score of 0.537, outperforming the average F1 score of all submitted runs (0.502). Run 2 utilized a linear kernel SVM with unigram features, augmented by features from MetaMap and CTAKES. Their deep learning model (Run 3), a bidirectional LSTM with GloVe embeddings and an attention layer, achieved an F1 score of 0.488. Overall, their runs performed above average for the task.

Limitations and Future Work

The authors acknowledge that their best performing model was a traditional machine learning approach, suggesting that further work is needed to fully leverage the potential of deep learning models for this task. They plan to further analyze the performance of deep learning models to improve ADE identification. The paper also implies that the complexity of distinguishing ADEs from indications remains a significant challenge.

Summary updated

BibTeX
@inproceedings{vydiswaran2019towards,
  author = {V.G.Vinod Vydiswaran and Grace Ganzel and Bryan Romas and Deahan Yu and Amy Austin and Neha Bhomia and Socheatha Chan and Stephanie Hall and Van Le and Aaron Miller and Olawunmi Oduyebo and Aulia Song and Radhika Sondhi and Danny Teng and Hao Tseng and Kim Vuong and Stephanie Zimmerman},
  title = {{Towards Text Processing Pipelines to Identify Adverse Drug Events-related Tweets: University of Michigan @ SMM4H 2019 Task 1}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure