Classification of Medication-Related Tweets Using Stacked Bidirectional LSTMs with Context-Aware Attention

Orest XherijaEMNLP 2018

TL;DR

This paper presents a system for classifying medication-related tweets using a stacked bidirectional LSTM with context-aware attention, achieving competitive results across four distinct subtasks of the SMM4H Shared Task. The model's strength lies in its adaptability and strong performance without task-specific tuning, demonstrating the power of attention mechanisms in understanding nuanced social media text.

What Problem Does This Paper Address?

The authors tackle the challenge of analyzing social media data for health-related insights, specifically focusing on medication-related tweets. This domain presents unique difficulties due to the inherent noise, informal language, spelling errors, and syntactic variations common in platforms like Twitter. Accurately classifying these tweets is crucial for understanding public health trends, drug reactions, and medication adherence. The SMM4H Shared Task aimed to push the boundaries of Natural Language Processing (NLP) in this area by proposing four distinct classification problems related to medication mentions, intake, adverse reactions, and vaccination behavior.

Methods

The core of the system is a variant of the Message-level Sentiment Analysis (MSA) model, originally developed for general Twitter sentiment analysis. This model employs a stacked bidirectional Long Short-Term Memory (BiLSTM) network. BiLSTMs are powerful for sequence data as they process information in both forward and backward directions, capturing context from both past and future words.

The key innovation is the context-aware attention mechanism. This mechanism allows the model to dynamically weigh the importance of different words in a tweet when making a classification decision. Instead of treating all words equally, attention helps the model focus on the most informative parts of the text, which is particularly useful for short, noisy social media posts where specific keywords can be highly indicative. The model uses word embeddings pre-trained on a massive Twitter corpus, which are crucial for understanding the vocabulary used in this domain.

Datasets

The paper evaluates the system on four subtasks from the 2018 Social Media Mining for Health Applications (SMM4H) Shared Task. These subtasks involved classifying tweets related to:

  • Subtask 1: Mention of drug or dietary supplement names.
  • Subtask 2: Description of medication intake (personal intake, suggested intake, or mention without intake indication).
  • Subtask 3: Mention of adverse drug reactions (ADR).
  • Subtask 4: Mention of vaccination behavior related to influenza vaccines.

The datasets were manually annotated by the organizers. A notable aspect is that the initial training sets did not contain the actual tweet text but rather tweet IDs, requiring participants to download the text. This process led to a reduction in the number of available tweets due to deletions or download issues.

Key Insights and Ideas

  • Adaptability of Attention-based LSTMs: The paper demonstrates that a sophisticated NLP model, like the BiLSTM with attention, can achieve strong performance across diverse classification tasks without task-specific fine-tuning. This highlights the generalizability of such architectures.
  • Importance of Pre-trained Embeddings: Using word embeddings trained on a large Twitter corpus significantly boosts performance, as it captures the specific language and nuances of social media text relevant to health.
  • Context is King in Short Text: The success of the attention mechanism underscores the critical role of understanding word context, especially in short, informal texts where meaning can be highly dependent on surrounding words.

Key Results

The system achieved competitive results across all four subtasks.

  • It outperformed the average performance of all submitted systems in Subtask 1 and Subtask 4.
  • Notably, it secured first place in Subtask 2 with a significant F1-score margin.
  • In Subtask 3, the system ranked 6th out of 9 participants.

Limitations and Future Work

The authors acknowledge that the model's performance in Subtask 3 might have been impacted by the decision to use a general model without specific tuning for that task, while other teams might have developed specialized systems.

Promising future directions include:

  • Ensembling Models: Investigating the benefits of combining multiple models, such as ensembles of word- and character-level attentive RNNs, to potentially improve performance further.
  • Multi-task Learning: Exploring multi-task learning approaches, where a single model is trained on multiple related tasks simultaneously. This could leverage shared information across the subtasks and potentially lead to better generalization and performance, especially given the overlap in lexicon and nature of the tasks.

Summary updated

BibTeX
@inproceedings{xherija2018classification,
  author = {Orest Xherija},
  title = {{Classification of Medication-Related Tweets Using Stacked Bidirectional LSTMs with Context-Aware Attention}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure