#phramacovigilance - Exploring Deep Learning Techniques for Identifying Mentions of Medication Intake from Twitter

Debanjan Mahata, Jasper Friedrichs, Hitkul, Rajiv Ratn ShaharXiv 2018 · arXiv

Authors: Debanjan Mahata, Jasper Friedrichs, Rajiv Ratn Shah, and Hitkul

Introduction

In the era of digital health, social media platforms like Twitter have emerged as invaluable resources for pharmacovigilance—the science and activities relating to the detection, assessment, understanding, and prevention of adverse effects or any other drug-related problems. While traditional pharmacovigilance relies on clinical reports and regulatory databases, Twitter offers a massive, real-time stream of user-generated content that can capture drug effects as they happen in the wild.

However, a significant hurdle remains: the "noise" of social media. A simple keyword search for a medication like "Aspirin" returns a mix of news articles, medical advice, and general discussions. For researchers to conduct meaningful studies on drug safety or patterns of abuse at an individual level, they must be able to distinguish between someone talking about a drug and someone actually consuming it. This paper presents a robust deep learning approach to solve this classification challenge.

Problem Definition and Dataset

The authors address the task of classifying tweets into three mutually exclusive categories:

  1. Personal Intake (Class 1): The user provides clear evidence of having consumed the medication (e.g., "Just took an Advil and feel better").
  2. Possible Intake (Class 2): The mention is ambiguous or suggests intent without confirmation (e.g., "I should have taken Advil on Friday").
  3. Non-intake (Class 3): The tweet mentions the drug in a general or third-person context (e.g., "Understand the causes of migraines #aspirin").

To train and evaluate their models, the researchers utilized the annotated dataset from the SMM4H (Social Media Mining for Health) 2017 shared task, consisting of 9,663 training tweets and 7,513 test tweets.

Methodology: From Shallow CNNs to Stacked Ensembles

The core of the proposed system is a specialized deep learning architecture designed to handle the unique linguistic characteristics of Twitter.

Shallow Convolutional Neural Networks (CNNs)

While deep architectures are popular in many AI domains, the authors found that "shallow" CNNs are particularly effective for short-form text like tweets. These models use convolutional layers with multiple filter sizes to capture local patterns (n-grams) across the sentence matrix.

Architecture of a single CNN model for sentence classification.

The input to these models consists of word embeddings—vector representations of words that capture semantic meaning. The study leveraged pre-trained embeddings specifically tuned for Twitter and health-related language, such as Godin Twitter Word2Vec and Drug Chatter Word2Vec.

The Stacked Ensemble Approach

The primary contribution of this work is the development of a Stacked Ensemble of Shallow CNNs. Recognizing that individual models can be sensitive to specific hyperparameters or idiosyncratic language, the authors employed a "wisdom of the crowd" strategy:

  1. Random Search: They performed an extensive random search over a vast hyperparameter space, including filter sizes, dropout rates, and batch sizes.
  2. Ensemble Formation: Hundreds of models were trained, and the top-performing ones were grouped into ensembles.
  3. Stacking: Finally, these ensembles were "stacked" to average their predictions, creating a more stable and robust final classifier.
The proposed stacked ensemble architecture combining multiple CNN models to produce a final prediction.

Results and Insights

The stacked ensemble approach yielded state-of-the-art results on the SMM4H benchmark. The best-performing configuration achieved a micro-averaged F-score of 0.693, outperforming previous top entries from institutions like UKNLP and NRC-Canada.

Key takeaways from the experimental results include:

  • Shallow is Effective: For 280-character messages, wide but shallow CNNs are better at capturing relevant context than deeper, more complex architectures that risk overfitting.
  • Robustness through Diversity: Averaging the predictions of the top 10 or 20 ensembles significantly mitigated the variance inherent in social media text, ensuring that no single slang term or typo could easily derail the classification.
  • Hyperparameter Importance: The performance was highly sensitive to choices like filter count and dropout probability, underscoring the value of the authors' exhaustive search process.

Conclusion and Future Directions

This research demonstrates that deep learning, when combined with robust ensembling techniques, can effectively bridge the gap between general social media discussion and actionable health data. By accurately identifying medication intake, public health researchers can better monitor drug safety and usage patterns in real-time.

Future work may involve scaling these models with even larger annotated datasets and exploring domain-specific embeddings trained from scratch on massive health-related corpora. Additionally, integrating sentiment analysis (affective computing) could provide deeper insights into the emotional states of users at the moment of medication intake.

Summary updated

Abstract

Mining social media messages for health and drug related information has received significant interest in pharmacovigilance research. Social media sites (e.g., Twitter), have been used for monitoring drug abuse, adverse reactions of drug usage and analyzing expression of sentiments related to drugs. Most of these studies are based on aggregated results from a large population rather than specific sets of individuals. In order to conduct studies at an individual level or specific cohorts, identifying posts mentioning intake of medicine by the user is necessary. Towards this objective, we train different deep neural network classification models on a publicly available annotated dataset and study their performances on identifying mentions of personal intake of medicine in tweets. We also design and train a new architecture of a stacked ensemble of shallow convolutional neural network (CNN) ensembles. We use random search for tuning the hyperparameters of the models and share the details of the values taken by the hyperparameters for the best learnt model in different deep neural network architectures. Our system produces state-of-the-art results, with a micro- averaged F-score of 0.693.
BibTeX
@misc{mahata2018phramacovigilance,
  author = {Debanjan Mahata and Jasper Friedrichs and Hitkul and Rajiv Ratn Shah},
  title = {{\#phramacovigilance - Exploring Deep Learning Techniques for Identifying Mentions of Medication Intake from Twitter}},
  year = {2018},
  eprint = {1805.06375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.06375},
}
Figure