Did you take the pill? - Detecting Personal Intake of Medicine from Twitter

Debanjan Mahata, Jasper Friedrichs, Rajiv Ratn Shah, Jing JiangarXiv 2018 · arXiv

  • Detecting Personal Intake of Medicine from Twitter

Abstract

Social media provides a rich source of data for pharmacovigilance, yet distinguishing between general drug mentions and actual personal intake remains a significant challenge. This paper presents a system using a stacked ensemble of shallow Convolutional Neural Networks (CNNs) to classify tweets into personal intake, possible intake, and non-intake. By aggregating predictions from multiple high-performing ensembles, the proposed InfyNLP system achieved state-of-the-art results in the SMM4H-2017 shared task, demonstrating the effectiveness of ensemble stacking for noisy, short-text health data.

Introduction

Pharmacovigilance—the monitoring of drug effects after they have been licensed for use—increasingly relies on social media data. However, aggregate mention counts are often misleading. A tweet mentioning "Aspirin" might refer to a news article or a hypothetical scenario rather than actual consumption. To accurately track drug efficacy and adverse reactions, researchers must isolate "personal intake." This task is complicated by the inherent noise of Twitter, characterized by slang, typos, and brevity. This study aims to classify tweets into three categories: personal medication intake, possible medication intake, and non-intake.

Methodology

The researchers developed InfyNLP, a system based on a Stacked Ensemble of Shallow Convolutional Neural Networks (CNNs). While deep architectures are common, shallow CNNs are often more effective for short-text classification as they can efficiently identify salient local features like specific word sequences regardless of their position in a tweet.

The core innovation lies in the "stacked ensemble" approach. The team performed 5-fold cross-validation to create initial ensembles and repeated this process 99 times using Random Search for hyperparameter optimization. The final prediction is an average of the top $K$ performing ensembles, which significantly reduces variance and enhances overall robustness.

Architecture of the Stacked Ensemble of Shallow CNNs *[Did you take the pill?

  • Detecting Personal Intake of Medicine from Twitter, page 3](/pdf/art_0838038ed05c4eab90d237f3923a87ce#page=3)*

Data and Preprocessing

The model was trained and tested on the SMM4H-2017 dataset, consisting of 9,663 training/development tweets and 7,513 test tweets. Preprocessing involved tokenization via the Spacy library. Notably, stopwords were retained, as small words like "I" or "my" are critical indicators of personal experience in health-related contexts. The system utilized 400-dimensional pre-trained word embeddings (Godin and Shin) specifically optimized for social media context.

Results and Discussion

The InfyNLP system ranked 1st out of 26 submissions at the SMM4H workshop, achieving a micro-averaged F1-score of 0.693 for the primary classes (Intake and Possible Intake). It demonstrated superior precision compared to other state-of-the-art models, including those utilizing attention mechanisms or SVMs.

A key finding was that stacking multiple ensembles consistently outperformed individual models. As shown in the performance analysis, the stacked ensemble maintains a more stable and higher evaluation score on test data compared to individual CNN ensembles, which exhibit higher variance.

Performance comparison between individual CNN ensembles and the stacked ensemble approach *[Did you take the pill?

  • Detecting Personal Intake of Medicine from Twitter, page 6](/pdf/art_0838038ed05c4eab90d237f3923a87ce#page=6)*

Furthermore, experimentation with filter sizes revealed that a size of 5 was generally optimal for capturing the linguistic context necessary to distinguish between actual and hypothetical intake. The model's success suggests it is learning nuanced cues, such as verb tense and pronoun usage, rather than relying solely on drug keywords.

Conclusion and Future Work

The success of the stacked shallow CNN approach provides a robust framework for individual-level pharmacovigilance. Future improvements could involve training embeddings on domain-specific medical corpora or incorporating multimodal data like emojis and images. Ultimately, such classifiers could enable real-time tracking of patient sentiment and psychological states following medication intake, offering insights beyond traditional clinical trials.

Summary updated

Abstract

Mining social media messages such as tweets, articles, and Facebook posts for health and drug related information has received significant interest in pharmacovigilance research. Social media sites (e.g., Twitter), have been used for monitoring drug abuse, adverse reactions of drug usage and analyzing expression of sentiments related to drugs. Most of these studies are based on aggregated results from a large population rather than specific sets of individuals. In order to conduct studies at an individual level or specific cohorts, identifying posts mentioning intake of medicine by the user is necessary. Towards this objective we develop a classifier for identifying mentions of personal intake of medicine in tweets. We train a stacked ensemble of shallow convolutional neural network (CNN) models on an annotated dataset. We use random search for tuning the hyper-parameters of the CNN models and present an ensemble of best models for the prediction task. Our system produces state-of-the-art result, with a micro-averaged F-score of 0.693. We believe that the developed classifier has direct uses in the areas of psychology, health informatics, pharmacovigilance and affective computing for tracking moods, emotions and sentiments of patients expressing intake of medicine in social media.
BibTeX
@misc{mahata2018did,
  author = {Debanjan Mahata and Jasper Friedrichs and Rajiv Ratn Shah and Jing Jiang},
  title = {{Did you take the pill? - Detecting Personal Intake of Medicine from Twitter}},
  year = {2018},
  eprint = {1808.02082},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.02082},
}
Figure