Detection of Adverse Drug Reaction in Tweets Using a Combination of Heterogeneous Word Embeddings

Segun Taofeek Aroyehun, Alexander GelbukhACL 2019

TL;DR

This paper tackles the challenging task of identifying adverse drug reactions (ADRs) in social media by combining multiple types of word embeddings (character, GloVe, and BERT) with a bidirectional LSTM. They achieved a competitive F1 score of 0.5209 on a highly imbalanced dataset, highlighting the significant contribution of character-level embeddings.

What Problem Does This Paper Address?

The paper addresses the critical need to automatically detect adverse drug reactions (ADRs) from social media, specifically tweets. This is crucial for public health surveillance, as social media platforms offer a vast, real-time source of patient experiences. However, this task is fraught with challenges:

  • Imbalanced Data: ADR reports are rare, making up only about 1% of the training data, which poses a significant hurdle for model training.
  • Distinguishing ADRs from Indications: Tweets can mention drugs and their effects, but it's vital to differentiate between a drug's intended use (indication) and an unintended, negative side effect (ADR).
  • Noisy Social Media Text: Tweets are often short, informal, and contain misspellings, abbreviations, and slang, making traditional natural language processing (NLP) techniques difficult to apply.

The authors aim to build a robust model that can accurately identify ADRs amidst this noise and imbalance.

Methods

The core of the proposed method is a combination of heterogeneous word embeddings fed into a bidirectional Long Short-Term Memory (LSTM) network. Here's a breakdown:

  • Heterogeneous Word Embeddings: The authors leverage three distinct types of word representations:
  • Character Embeddings: These are learned representations of individual characters within a word, combined to form a word embedding. This approach is trained jointly with the model and is particularly effective at handling misspellings and abbreviations common in social media, thus mitigating the "out-of-vocabulary" problem.
  • GloVe (Global Vectors for Word Representation): A pre-trained, non-contextual word embedding model trained on a large Twitter corpus. This is expected to capture general Twitter language patterns.
  • BERT (Bidirectional Encoder Representations from Transformers): A powerful, pre-trained contextual word embedding model. BERT's representations are dynamic, changing based on the surrounding words in a sentence, allowing for a deeper understanding of word meaning.
  • Concatenation and Projection: These three embedding types for each word are concatenated, creating a rich, multi-faceted representation. This combined vector is then projected into a lower dimension (256) using a linear layer, acting as a distillation or fine-tuning step.
  • Bidirectional LSTM: The projected representations are fed into a bidirectional LSTM, which processes the tweet sequence in both forward and backward directions. This allows the model to capture sequential dependencies and context within the tweet.
  • Classification: Finally, a dense layer acts as a classifier to determine whether a tweet reports an ADR.

Datasets

  • Training, Validation, and Test Data: The experiments were conducted using data from the 2019 Social Media Mining for Healthcare Applications (SMM4H) shared task. The dataset is described as having approximately 1% positive examples (ADRs) and 99% negative examples.
  • Training set: 24,202 examples
    • Development (validation) set: 6,051 examples
    • Test set: 4,575 examples

Key Insights and Ideas

  • Character Embeddings are Crucial for Social Media: The paper strongly suggests that character-level embeddings are vital for handling the noisy and informal nature of social media text, outperforming other representations in their ablation studies.
  • Combination is Key, but Nuanced: While combining embeddings generally helps, the specific contribution of each type varies. The authors found that removing character embeddings led to a significant performance drop, whereas removing BERT or GloVe surprisingly improved performance in some cases.
  • Fine-tuning Layer's Role is Unclear: The authors observed a minimal performance drop when removing the fine-tuning layer, suggesting its benefit might be less than expected or that its parameters require further tuning.
  • BERT's Domain Mismatch: The improved performance when BERT was removed hints at a potential domain mismatch between the general-domain BERT pre-training and the specific healthcare-related tweets.

Key Results

  • Best F1 Score: The proposed model achieved an F1 score of 0.5209 on the test set, which ranked above the average score of all participants in the SMM4H task.
  • Ablation Study Findings:
    • Removing character embeddings resulted in a performance drop of 0.0238.
  • Removing BERT representation led to an improvement of 0.0025.
  • Removing GloVe embedding resulted in an improvement of 0.0005.
  • Submission Performance:
    • Submission 1 (best model on validation set): F1 = 0.5167
    • Submission 2 (model at last epoch): F1 = 0.5209

Limitations and Future Work

  • Fine-tuning Layer Uncertainty: The authors are unsure about the exact role and benefit of the fine-tuning layer, suggesting further investigation into its parameters and approaches.
  • BERT Domain Mismatch: The observed performance improvement upon BERT removal indicates a need to explore domain-specific BERT models or alternative contextual embeddings for this task.
  • Future Directions:
    • Investigate other fine-tuning or distillation techniques.
  • Explore parameter tuning for the fine-tuning layer's size.
  • Examine the impact of tweet normalization.
  • Consider identifying "usage expressions" as an auxiliary task to potentially improve ADR detection.

Summary updated

BibTeX
@inproceedings{aroyehun2019detection,
  author = {Segun Taofeek Aroyehun and Alexander Gelbukh},
  title = {{Detection of Adverse Drug Reaction in Tweets Using a Combination of Heterogeneous Word Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure