Abstract
Detecting Adverse Drug Effects (ADEs) in social media is a critical task for pharmacovigilance, yet it is complicated by the informal nature of tweets and extreme class imbalance. This paper introduces Template Augmented Task Adaptive Pre-training (TAPT), a method that appends descriptive prompt templates to task-specific data during a secondary pre-training phase. Using BERTweet as a foundation, the authors demonstrate that this approach significantly improves the model's ability to identify ADEs in the noisy Twitter environment, outperforming standard fine-tuning and unsupervised pre-training methods.
Introduction
Social media platforms like Twitter serve as a valuable resource for monitoring drug effects post-licensing. However, identifying ADEs in tweets is notoriously difficult due to the brevity of posts, the use of slang, and the "extremely imbalanced" nature of the data—where only approximately 7% of tweets in the SMM4H 2022 dataset mention an ADE. Standard large language models (LLMs) often struggle with this niche domain. While Task Adaptive Pre-training (TAPT) helps, the limited size of specialized datasets often restricts its impact.
Methodology
The researchers at KU Leuven utilized BERTweet, a model pre-trained on 850 million tweets, and applied two primary strategies to bridge the gap to ADE detection:
- Template Augmentation: The core novelty involves generating multiple training sentence variants by appending prompt templates (e.g., "It contains an ADE", "ADE is present"). This reinforces label information during the pre-training phase.
- Pre-training Objectives: The study compared Masked Language Modeling (MLM) and Simple Contrastive Learning (SimCSE). MLM involves predicting hidden tokens, while SimCSE uses dropout masks to force the model to recognize semantic similarities.
- Class Imbalance Mitigation: Random oversampling was employed to increase the minority class (ADE mentions) to 10,000 samples, ensuring a more balanced distribution for fine-tuning.
Experimental Setup
The study utilized the SMM4H 2022 Task 1a dataset, consisting of 18,000 training tweets, 915 validation tweets, and 10,000 test tweets. The dataset is characterized by binary labels (ADE vs. noADE) and a high degree of imbalance.
Results and Discussion
The "Template Augmented" approach demonstrated clear superiority over standard fine-tuning methods.
Validation Performance
On the validation set, the mlm_aug model achieved an F1 score of 0.7941, significantly outperforming the base BERTweet model's F1 of 0.7143. This indicates that the template-based hints effectively help the model cluster task-specific representations.
Test Performance and Post-Evaluation
The official competition submission reached an F1 of 0.433. However, post-evaluation experiments using additional templates pushed the test F1 score to 0.584, exceeding the mean results of other participants (0.562).
Key Insights
- Templates as Supervised Signals: Adding templates during pre-training acts as a "hint," helping the model cluster representations more effectively than unsupervised TAPT.
- MLM vs. SimCSE: MLM with templates (
mlm_aug) outperformed the contrastive approach. The authors suggest that masking tokens in the presence of templates allows the model to better encapsulate the task distribution.
Conclusion and Future Work
Despite the improvements, a significant "generalization gap" remains, as evidenced by the drop from validation (~0.79 F1) to test (~0.43-0.58 F1) performance. This suggests potential overfitting to the training distribution. Future work will explore back-translation for data augmentation and the incorporation of external medical knowledge bases to improve the recognition of rare drug names and terminology.
Summary updated