Abstract
This paper presents a description of our system in SMM4H-2022, where we participated in task 1a, task 4, and task 6 to task 10. There are three main challenges in SMM4H-2022, namely the domain shift problem, the prediction bias due to category imbalance, and the noise in informal text. In this paper, we propose a unified framework for the classification and named entity recognition tasks to solve the challenges, and it can be applied to both English and Spanish scenarios. The results of our system are higher than the median F1-scores for 7 tasks and significantly exceed the F1-scores for 6 tasks. The experimental results demonstrate the effectiveness of our system.
Introduction
Extracting medical insights from platforms like Twitter and Reddit is notoriously difficult. Unlike clinical records, social media text is riddled with slang, emojis, misspellings, and colloquialisms. The CASIA team identifies three primary technical challenges in this domain:
- Domain Shift: Large language models (LLMs) like RoBERTa are trained on general text, creating a performance gap when applied to niche medical discussions.
- Category Imbalance: Positive health reports (e.g., drug side effects) are rare, often comprising only 7–11% of datasets, leading to model bias.
- Noise: Usernames, URLs, and emojis distract models from actual semantic content.
Methodology
The authors developed a multi-stage pipeline to handle these issues, applicable to both English and Spanish scenarios.
Data Processing and Pre-training
- Data Cleaning & Balancing: URLs and usernames were stripped, and emojis were replaced with text descriptions. To fix imbalance, the team used oversampling, duplicating minority class samples to reach specific ratios (1:1 or 0.5:1).
- Domain-Adaptive Pre-training (DAP): Models (RoBERTa for English, BETO for Spanish) were "warmed up" by continuing pre-training on large sets of unlabeled health-related social media posts.
Advanced Fine-tuning
- Child-Tuning: Instead of updating all parameters, this strategy identifies and updates only a "child network" of the most relevant parameters, preventing overfitting on small datasets.
- Adversarial Training (FGM): The Fast Gradient Method was used to inject small perturbations into word embeddings, teaching the model to ignore minor noise.
Key Results and Insights
The CASIA system was highly competitive, exceeding the median F1-score for 7 out of 8 tasks.
- Task 1a (Adverse Events): Achieved an F1-score of 0.8095, 7% higher than the participant mean.
- Task 4 (Age Reporting): Reached an F1 of 0.929, exceeding the median by 4.7%.
- Spanish NER (Task 10): A fused ensemble of 15 models achieved a final Test F1-score of 0.891.
Experimental results highlighted that Child-Tuning consistently outperformed standard AdamW optimization in imbalanced tasks. Furthermore, the team found that a perfect 1:1 class ratio wasn't always optimal; for several tasks, an oversampling ratio of 0.5:1 yielded the highest F1-scores, suggesting that over-duplication can introduce its own biases.
Conclusion
The CASIA framework demonstrates that a unified "recipe"—DAP followed by robust fine-tuning—is effective for both classification and NER across multiple languages. Future work will explore incorporating Medical Knowledge Graphs to link social media mentions to structured medical databases like UMLS, providing deeper semantic context.
Summary updated