Abstract
Identifying medication mentions in social media text is a critical task for public health and pharmacovigilance, yet it is hindered by extreme class imbalance. This paper presents a hybrid deep learning pipeline developed for the SMM4H 2020 Task 1. By utilizing BERT as a contextual sentence encoder combined with a Bidirectional Long Short-Term Memory (BiLSTM) classifier and an intermediate "warm-up" training phase on balanced data, the proposed system achieves an F1 score of 0.7164, significantly outperforming the competition average.
Introduction
Twitter serves as a vast repository of patient-generated data, but extracting specific health signals like medication mentions is challenging. In the SMM4H 2020 Task 1 dataset, only 0.2% of nearly 70,000 tweets contained positive mentions. Standard machine learning models often struggle with such high imbalance, frequently defaulting to majority-class predictions to maintain nominal accuracy. This research aims to develop a robust model capable of maintaining high precision and recall despite the rarity of the target signal.
Methodology
The authors proposed a three-stage approach designed to optimize feature extraction and classification for imbalanced data:
- Preprocessing: Using the
Ekphrasislibrary, social media noise was normalized. This involved replacing usernames and URLs with generic tokens, segmenting hashtags, and performing spell correction. - Preliminary Fine-tuning: To mitigate the effects of imbalance, the BERT model was first fine-tuned on an auxiliary, balanced dataset from SMM4H 2018. This "warm-up" phase allowed the model to learn medication-related linguistic patterns before exposure to the primary imbalanced dataset.
- Hybrid Architecture: The system utilized BERT as a sentence encoder, feeding its internal layer activations into a BiLSTM network. The BiLSTM was chosen for its ability to capture sequential context, which is vital for understanding short, informal text like tweets.
A critical component of the experiment was comparing "frozen" vs. "unfrozen" BERT parameters. In the unfrozen configuration, BERT weights were updated during the final training phase, allowing for specialized transfer learning.
Experimental Setup and Datasets
- Primary Dataset (SMM4H 2020 Task 1): 69,272 training tweets (0.2% positive) and 29,687 test examples.
- Auxiliary Dataset (SMM4H 2018 Task 1): 2,367 tweets with a balanced distribution, used for initial fine-tuning.
Results and Analysis
The hybrid BERT-BiLSTM approach demonstrated superior performance compared to standard baselines. Internal validation highlighted two key technical insights: the impact of unfreezing parameters and the choice of hidden layers for feature extraction.
As shown in Figure 1, the unfrozen model achieved its best performance (F1: 0.88) when concatenating the activations of the final four hidden layers of BERT. This suggests that combining different levels of semantic information provides a richer representation for the BiLSTM.
In contrast, Figure 2 illustrates the performance of the frozen model. While still effective, the peak F1 score (0.81) was lower than the unfrozen counterpart, and the optimal strategy shifted to using only the last hidden layer. These results confirm that allowing BERT parameters to adapt to the specific task domain significantly improves classification accuracy.
On the official competition test set, the system achieved an F1 score of 0.7164, a precision of 0.8421, and a recall of 0.6234, surpassing the participant mean F1 of 0.6646.
Conclusion
The study demonstrates that a combination of intermediate training on balanced data and a hybrid BERT-BiLSTM architecture can effectively address extreme class imbalance in social media text classification. While the model achieved high precision, the lower recall (0.62) indicates that capturing all medication mentions remains a challenge. Future work will explore advanced data augmentation and loss-weighting techniques to further improve recall in highly imbalanced environments.
Summary updated