TL;DR
This paper presents Transformer-based models for classifying health-related information and detecting adverse drug effect (ADE) mentions in social media. The authors achieved strong results, including first place in one task and outperforming the median in most others, by leveraging pre-trained models and employing techniques like oversampling and ensemble methods.
What Problem Does This Paper Address?
Social media platforms are a rich source of information about people's health experiences, including adverse drug effects (ADEs). However, extracting this information is challenging due to the informal, noisy, and often ungrammatical nature of user-generated content. To facilitate this extraction, the SMM4H (Social Media Mining for Health Applications) shared tasks were organized. This paper focuses on the SMM4H 2021 shared tasks, specifically addressing six classification tasks and one span detection task related to health information on social media. The core challenge is to build robust models that can accurately understand and process this complex data to identify and categorize health-related mentions.
Methods
The authors developed two distinct systems: one for classification tasks and another for span detection. Both systems are built upon pre-trained Transformer-based models, a standard and highly effective approach in Natural Language Processing (NLP).
For classification, they experimented with several Transformer variants:
- BERTweet (BT): Pre-trained on English tweets.
- Bio_Clinical BERT (CL): Pre-trained on biomedical research papers and clinical notes.
- RoBERTaBase (RBB) and RoBERTaLarge (RBL): Pre-trained on general text like English Wikipedia.
They also employed:
- Preprocessing: A tool called
preprocess-twitterwas used to clean tweets by lowercasing, normalizing numbers, URLs, usernames, and handling hashtags and emoticons. - Oversampling: For imbalanced datasets (Tasks 1a, 3a, and 5), they used random oversampling with replacement to balance the class distribution.
- Ensemble Modeling: To further improve performance, they averaged the output probabilities of different models and selected the class with the highest combined probability.
For span detection, they experimented with several Transformer models, including BioBERT, SciBERT, BERTweet, BERT Base, and BERT Large. They also explored undersampling for the imbalanced dataset by dividing the negative class into subsets.
Datasets
The primary datasets used are from the SMM4H 2021 shared tasks. These include tweets and drug reviews annotated for various health-related information. Specifically:
- Task 1a: Classification of adverse effect mentions in English tweets.
- Task 3a & 3b: Classification of medication regimen changes in tweets and drug reviews.
- Task 4: Classification of tweets reporting adverse pregnancy outcomes.
- Task 5: Classification of tweets reporting potential COVID-19 cases.
- Task 6: Classification of COVID-19 tweets containing symptoms.
- Task 1b: ADE span detection.
The authors split the training data into training (TRN) and validation (TRN_VAL) sets (90/10 ratio) and evaluated on the official validation set (VAL) released by the organizers.
Key Insights and Ideas
- Domain-specific pre-training matters, but general pre-training can also be effective: While Bio_Clinical BERT underperformed, BERTweet and RoBERTa models showed strong performance, suggesting that models pre-trained on social media data (BERTweet) or general text (RoBERTa) are highly competitive for health-related social media tasks.
- Oversampling helps with imbalanced classification: For imbalanced datasets, oversampling the minority class was a crucial step to improve performance.
- Ensembling boosts performance: Combining multiple models through ensembling generally led to better results than individual models, except for Task 1a.
- ADE description variability is a major challenge: For Task 1a, the significant drop in performance from the validation set to the test set was attributed to the limited generalizability of models to capture semantic similarities in ADE descriptions, even when the underlying ADE is the same. This highlights the difficulty of handling diverse ways people describe health issues.
- BERT Large excels in span detection: For Task 1b, BERT Large demonstrated superior performance, particularly in recall and F1-score, indicating its effectiveness for identifying specific text spans.
- Undersampling can be detrimental: For span detection, undersampling the negative class led to a significant performance drop, suggesting that the amount of negative data is critical for this task.
Key Results
- Overall Performance: The authors' submissions were above the median F1-scores in most classification tasks.
- Task 4: Achieved first place in the competition.
- Task 1b (Span Detection): Achieved a 7% higher F1-score than the median.
- Task 3a: Achieved an 18% higher F1-score compared to a baseline model.
- Task 1a Analysis: While models performed well on the internal validation set, a significant performance drop was observed on the official test set, suggesting issues with ADE description variability.
Limitations and Future Work
- Generalizability of ADE descriptions: The paper identifies the challenge of models not generalizing well to different expressions of the same ADE, particularly in Task 1a. Future work will focus on improving this generalizability.
- Undersampling heuristics: For span detection, the authors acknowledge that more advanced heuristics for selecting negative examples during undersampling could be explored.
- Model size and domain impact: While they experimented with different model sizes and pre-training domains, further investigation into how these factors specifically influence performance on various health-related social media tasks could be beneficial.
Summary updated