TL;DR
This paper presents the BIGODM system for automatically classifying Twitter posts about adverse drug reactions (ADRs). They achieved competitive results by employing a vote-based under-sampling ensemble (VUE) of linear Support Vector Machines (SVMs) trained on a combined dataset from previous SMM4H shared tasks.
What Problem Does This Paper Address?
The paper tackles the challenge of automatically identifying mentions of adverse drug reactions (ADRs) within the noisy and informal text of social media, specifically Twitter. This is crucial for public health surveillance, allowing for early detection of potential drug side effects. The core problem lies in the inherent imbalance of such data – negative mentions (no ADR) far outweigh positive ones (ADR mentioned) – and the difficulty of extracting meaningful signals from short, often ungrammatical tweets.
Methods
The BIGODM system utilizes a Vote-Based Under-sampling Ensemble (VUE) technique. This approach addresses data imbalance by creating multiple SVM classifiers. Each classifier is trained on a subset of the majority class (negative tweets) combined with all instances of the minority class (positive tweets). The final prediction is determined by a majority vote across these individual classifiers.
The system employs linear Support Vector Machines (SVMs) as the base classifier. Feature engineering is a key component, including:
- Bag-of-word features: Unigrams and bigrams with TF-IDF weighting.
- Domain knowledge features: Binary flags indicating the presence of ADR or drug mentions, identified using a pre-existing ADR mention recognizer.
- Negation features: Flags to detect if an ADR mention is negated, using the NegEx algorithm.
- Word embedding features: A 400-dimensional vector derived from pre-trained tweet word embeddings.
Preprocessing steps include replacing URLs, dosages, and Twitter-specific characters, tokenizing tweets, correcting spelling errors with Hunspell, lowercasing, and stemming.
Datasets
The system was trained and evaluated on datasets from the ACL 2019 Social Media Mining for Health Applications (SMM4H) shared task 1. Specifically, they used:
- The SMM4H 2019 training and validation sets.
- The AMIA-SMM4H 2017 corpus.
These datasets were merged to create a larger training corpus. The SMM4H 2019 training and validation sets contained 24,861 and 5,000 tweets respectively, though only 2,887 validation tweets were downloadable. The SMM4H 2017 corpus added 11,564 tweets. After merging and deduplication, the combined corpus had 3,423 positive and 31,858 negative tweets, resulting in an imbalance ratio of 9.3.
Key Insights and Ideas
- Ensemble methods are crucial for imbalanced data: The VUE technique proved effective in handling the severe class imbalance, leading to better performance than a baseline model without such techniques.
- Merged datasets can be beneficial, but not always: While using a merged corpus (2017+2019) with the VUE model was the best performing configuration in the shared task, an interesting finding was that the baseline model (without VUE) performed better than VUE on this merged corpus. This suggests that the VUE's under-sampling might have removed too much valuable information from the combined dataset.
- Feature engineering remains important: The combination of bag-of-words, domain knowledge, negation, and word embeddings provided a rich set of features for the SVM classifiers.
Key Results
The BIGODM system achieved an F-measure of 0.551 on the test sets, which was approximately 5% higher than the average F-scores of the 16 participating teams. This indicates a strong performance in the shared task. The paper also presents results from 10-fold cross-validation on the training set, showing the performance of VUE and baseline models across different corpora (2019, 2017, and 2017+2019). The VUE model generally showed better recall but lower precision compared to the baseline.
Limitations and Future Work
The authors acknowledge an interesting finding where the baseline model outperformed the VUE model on the merged corpus. They plan to conduct error analysis to understand this phenomenon better. Future work will also involve investigating other advanced imbalance techniques developed in their previous research to potentially improve performance further.
Summary updated