KUL@SMM4H'22: Template Augmented Adaptive Pre-training for Tweet Classification

Sumam Francis, Marie-Francine MoensSMM4H @ COLING 2022 · OpenReview

Abstract

Detecting Adverse Drug Effects (ADEs) in social media is a critical task for pharmacovigilance, yet it is complicated by the informal nature of tweets and extreme class imbalance. This paper introduces Template Augmented Task Adaptive Pre-training (TAPT), a method that appends descriptive prompt templates to task-specific data during a secondary pre-training phase. Using BERTweet as a foundation, the authors demonstrate that this approach significantly improves the model's ability to identify ADEs in the noisy Twitter environment, outperforming standard fine-tuning and unsupervised pre-training methods.

Introduction

Social media platforms like Twitter serve as a valuable resource for monitoring drug effects post-licensing. However, identifying ADEs in tweets is notoriously difficult due to the brevity of posts, the use of slang, and the "extremely imbalanced" nature of the data—where only approximately 7% of tweets in the SMM4H 2022 dataset mention an ADE. Standard large language models (LLMs) often struggle with this niche domain. While Task Adaptive Pre-training (TAPT) helps, the limited size of specialized datasets often restricts its impact.

Methodology

The researchers at KU Leuven utilized BERTweet, a model pre-trained on 850 million tweets, and applied two primary strategies to bridge the gap to ADE detection:

  1. Template Augmentation: The core novelty involves generating multiple training sentence variants by appending prompt templates (e.g., "It contains an ADE", "ADE is present"). This reinforces label information during the pre-training phase.
  2. Pre-training Objectives: The study compared Masked Language Modeling (MLM) and Simple Contrastive Learning (SimCSE). MLM involves predicting hidden tokens, while SimCSE uses dropout masks to force the model to recognize semantic similarities.
  3. Class Imbalance Mitigation: Random oversampling was employed to increase the minority class (ADE mentions) to 10,000 samples, ensuring a more balanced distribution for fine-tuning.

Experimental Setup

The study utilized the SMM4H 2022 Task 1a dataset, consisting of 18,000 training tweets, 915 validation tweets, and 10,000 test tweets. The dataset is characterized by binary labels (ADE vs. noADE) and a high degree of imbalance.

Results and Discussion

The "Template Augmented" approach demonstrated clear superiority over standard fine-tuning methods.

Validation Performance

On the validation set, the mlm_aug model achieved an F1 score of 0.7941, significantly outperforming the base BERTweet model's F1 of 0.7143. This indicates that the template-based hints effectively help the model cluster task-specific representations.

Validation performance comparison showing the effectiveness of template-augmented MLM.

Test Performance and Post-Evaluation

The official competition submission reached an F1 of 0.433. However, post-evaluation experiments using additional templates pushed the test F1 score to 0.584, exceeding the mean results of other participants (0.562).

Test set performance and post-evaluation improvements.

Key Insights

  • Templates as Supervised Signals: Adding templates during pre-training acts as a "hint," helping the model cluster representations more effectively than unsupervised TAPT.
  • MLM vs. SimCSE: MLM with templates (mlm_aug) outperformed the contrastive approach. The authors suggest that masking tokens in the presence of templates allows the model to better encapsulate the task distribution.

Conclusion and Future Work

Despite the improvements, a significant "generalization gap" remains, as evidenced by the drop from validation (~0.79 F1) to test (~0.43-0.58 F1) performance. This suggests potential overfitting to the training distribution. Future work will explore back-translation for data augmentation and the incorporation of external medical knowledge bases to improve the recognition of rare drug names and terminology.

Summary updated

BibTeX
@inproceedings{francis2022kul,
  author = {Sumam Francis and Marie-Francine Moens},
  title = {{KUL@SMM4H'22: Template Augmented Adaptive Pre-training for Tweet Classification}},
  booktitle = {SMM4H @ COLING},
  year = {2022},
  url = {https://openreview.net/forum?id=KPgaKkRTwB},
}
Figure