Deep Learning for Identification of Adverse Effect Mentions In Twitter Data

Paul Barry, Ozlem UzunerACL 2019

TL;DR

This paper presents a deep learning system for automatically identifying adverse drug effect mentions in Twitter data. Their approach combines character-level and token-level embeddings with a bidirectional LSTM and Conditional Random Field, achieving strong performance on a challenging shared task.

What Problem Does This Paper Address?

Identifying adverse drug effects (AEs) from social media is crucial for public health surveillance. However, tweets are short, noisy, and often use informal language, making it difficult for automated systems to accurately pinpoint AE mentions. This paper tackles the challenge of accurately extracting these mentions from Twitter data, a task that requires handling variations in language, abbreviations, and multi-word expressions that constitute an AE. The SMM4H (Social Media Mining for Health Applications) shared task specifically aims to advance this capability.

Methods

The authors developed a deep learning system built upon a bidirectional Long Short-Term Memory (bi-LSTM) architecture. The core innovation lies in its multi-layered embedding strategy:

  • Character-Level Embeddings: A bi-LSTM processes individual characters within words to generate embeddings. This helps capture sub-word information and handle misspellings or abbreviations.
  • Token-Level Embeddings: Pre-trained GloVe embeddings are used as a starting point, and then further fine-tuned using ELMo.
  • Character-Enhanced Token Embeddings: The character-level embeddings are concatenated with the token-level embeddings to create a richer representation of each word.
  • Stacked Bi-LSTMs: These character-enhanced token embeddings are fed into a second bi-LSTM layer. This layer processes the sequence of words, with both forward and backward passes contributing to the prediction.
  • Conditional Random Field (CRF): The output of the bi-LSTM is then passed to a CRF layer, which optimizes the sequence of predicted labels, ensuring more coherent and accurate span identification.

Preprocessing steps included consolidating overlapping AE spans, replacing Twitter handles with "@person," removing URLs, and treating hashtags as regular words.

Datasets

  • Training Data: 2,367 unique tweets, with 1,212 positive examples (containing AE mentions) and 1,155 negative examples.
  • Evaluation Data: 1,000 tweets, with 500 positive and 500 negative examples.
  • Internal Split: The training dataset was further split into 1,657 tweets for training, 355 for validation, and 355 for testing to tune hyperparameters.

Key Insights and Ideas

  • Character-level information is vital: The inclusion of character-level embeddings significantly improved the model's ability to handle the nuances of social media text, such as misspellings and abbreviations, which are common in AE mentions.
  • Preprocessing has a high impact: Noise reduction techniques, like removing URLs and standardizing Twitter handles, proved more impactful on performance than extensive hyperparameter tuning.
  • Tokenizer choice matters: The choice of tokenizer had a notable effect on performance, with the NLTK's Word Punct tokenizer yielding the best results when special characters were retained.
  • Contextual understanding is challenging: The error analysis revealed that while words strongly associated with AEs are often identified, the model sometimes struggles when these words appear in contexts unrelated to medication side effects, suggesting a need for deeper contextual understanding.

Key Results

The system achieved strong performance on the SMM4H shared task:

  • Relaxed F1 Score: 59.7 (beating the task average by 5.9)
  • Strict F1 Score: 40.7 (beating the task average by 9.0)
  • The best results were obtained using the NLTK's Word Punct tokenizer with special characters kept.
  • Training the model was relatively fast, taking 19 minutes and 22 seconds on a 16-core CPU.

Limitations and Future Work

  • Contextual Errors: The model sometimes overemphasizes specific words and underemphasizes their context, leading to errors when AE-associated words appear in non-AE contexts.
  • Future Work Suggestions:
    • Explore larger corpora with more negative examples of AE-associated words.
  • Investigate additional LSTM layers in the label prediction layer.
  • Incorporate more recent word embedding algorithms.

Summary updated

BibTeX
@inproceedings{barry2019deep,
  author = {Paul Barry and Ozlem Uzuner},
  title = {{Deep Learning for Identification of Adverse Effect Mentions In Twitter Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure