- Detecting Personal Intake of Medicine from Twitter
Abstract
Social media provides a rich source of data for pharmacovigilance, yet distinguishing between general drug mentions and actual personal intake remains a significant challenge. This paper presents a system using a stacked ensemble of shallow Convolutional Neural Networks (CNNs) to classify tweets into personal intake, possible intake, and non-intake. By aggregating predictions from multiple high-performing ensembles, the proposed InfyNLP system achieved state-of-the-art results in the SMM4H-2017 shared task, demonstrating the effectiveness of ensemble stacking for noisy, short-text health data.
Introduction
Pharmacovigilance—the monitoring of drug effects after they have been licensed for use—increasingly relies on social media data. However, aggregate mention counts are often misleading. A tweet mentioning "Aspirin" might refer to a news article or a hypothetical scenario rather than actual consumption. To accurately track drug efficacy and adverse reactions, researchers must isolate "personal intake." This task is complicated by the inherent noise of Twitter, characterized by slang, typos, and brevity. This study aims to classify tweets into three categories: personal medication intake, possible medication intake, and non-intake.
Methodology
The researchers developed InfyNLP, a system based on a Stacked Ensemble of Shallow Convolutional Neural Networks (CNNs). While deep architectures are common, shallow CNNs are often more effective for short-text classification as they can efficiently identify salient local features like specific word sequences regardless of their position in a tweet.
The core innovation lies in the "stacked ensemble" approach. The team performed 5-fold cross-validation to create initial ensembles and repeated this process 99 times using Random Search for hyperparameter optimization. The final prediction is an average of the top $K$ performing ensembles, which significantly reduces variance and enhances overall robustness.
*[Did you take the pill?
- Detecting Personal Intake of Medicine from Twitter, page 3](/pdf/art_0838038ed05c4eab90d237f3923a87ce#page=3)*
Data and Preprocessing
The model was trained and tested on the SMM4H-2017 dataset, consisting of 9,663 training/development tweets and 7,513 test tweets. Preprocessing involved tokenization via the Spacy library. Notably, stopwords were retained, as small words like "I" or "my" are critical indicators of personal experience in health-related contexts. The system utilized 400-dimensional pre-trained word embeddings (Godin and Shin) specifically optimized for social media context.
Results and Discussion
The InfyNLP system ranked 1st out of 26 submissions at the SMM4H workshop, achieving a micro-averaged F1-score of 0.693 for the primary classes (Intake and Possible Intake). It demonstrated superior precision compared to other state-of-the-art models, including those utilizing attention mechanisms or SVMs.
A key finding was that stacking multiple ensembles consistently outperformed individual models. As shown in the performance analysis, the stacked ensemble maintains a more stable and higher evaluation score on test data compared to individual CNN ensembles, which exhibit higher variance.
*[Did you take the pill?
- Detecting Personal Intake of Medicine from Twitter, page 6](/pdf/art_0838038ed05c4eab90d237f3923a87ce#page=6)*
Furthermore, experimentation with filter sizes revealed that a size of 5 was generally optimal for capturing the linguistic context necessary to distinguish between actual and hypothetical intake. The model's success suggests it is learning nuanced cues, such as verb tense and pronoun usage, rather than relying solely on drug keywords.
Conclusion and Future Work
The success of the stacked shallow CNN approach provides a robust framework for individual-level pharmacovigilance. Future improvements could involve training embeddings on domain-specific medical corpora or incorporating multimodal data like emojis and images. Ultimately, such classifiers could enable real-time tracking of patient sentiment and psychological states following medication intake, offering insights beyond traditional clinical trials.
Summary updated