CASIA@SMM4H'22: A Uniform Health Information Mining System for Multilingual Social Media Texts

Jia Fu, Sirui Li, Hui Ming Yuan, Zhucong Li, Zhen Gan, Yubo Chen, Kang Liu, Jun Zhao, Shengping LiuSMM4H @ COLING 2022 · OpenReview

Abstract

This paper presents a description of our system in SMM4H-2022, where we participated in task 1a, task 4, and task 6 to task 10. There are three main challenges in SMM4H-2022, namely the domain shift problem, the prediction bias due to category imbalance, and the noise in informal text. In this paper, we propose a unified framework for the classification and named entity recognition tasks to solve the challenges, and it can be applied to both English and Spanish scenarios. The results of our system are higher than the median F1-scores for 7 tasks and significantly exceed the F1-scores for 6 tasks. The experimental results demonstrate the effectiveness of our system.

Introduction

Extracting medical insights from platforms like Twitter and Reddit is notoriously difficult. Unlike clinical records, social media text is riddled with slang, emojis, misspellings, and colloquialisms. The CASIA team identifies three primary technical challenges in this domain:

  1. Domain Shift: Large language models (LLMs) like RoBERTa are trained on general text, creating a performance gap when applied to niche medical discussions.
  2. Category Imbalance: Positive health reports (e.g., drug side effects) are rare, often comprising only 7–11% of datasets, leading to model bias.
  3. Noise: Usernames, URLs, and emojis distract models from actual semantic content.

Methodology

The authors developed a multi-stage pipeline to handle these issues, applicable to both English and Spanish scenarios.

The CASIA system architecture, illustrating the progression from raw data to final results through domain-adaptive pre-training and specialized fine-tuning.

Data Processing and Pre-training

  • Data Cleaning & Balancing: URLs and usernames were stripped, and emojis were replaced with text descriptions. To fix imbalance, the team used oversampling, duplicating minority class samples to reach specific ratios (1:1 or 0.5:1).
  • Domain-Adaptive Pre-training (DAP): Models (RoBERTa for English, BETO for Spanish) were "warmed up" by continuing pre-training on large sets of unlabeled health-related social media posts.

Advanced Fine-tuning

  • Child-Tuning: Instead of updating all parameters, this strategy identifies and updates only a "child network" of the most relevant parameters, preventing overfitting on small datasets.
  • Adversarial Training (FGM): The Fast Gradient Method was used to inject small perturbations into word embeddings, teaching the model to ignore minor noise.

Key Results and Insights

The CASIA system was highly competitive, exceeding the median F1-score for 7 out of 8 tasks.

Comparison of system performance against mean and median F1-scores across multiple SMM4H tasks.
  • Task 1a (Adverse Events): Achieved an F1-score of 0.8095, 7% higher than the participant mean.
  • Task 4 (Age Reporting): Reached an F1 of 0.929, exceeding the median by 4.7%.
  • Spanish NER (Task 10): A fused ensemble of 15 models achieved a final Test F1-score of 0.891.

Experimental results highlighted that Child-Tuning consistently outperformed standard AdamW optimization in imbalanced tasks. Furthermore, the team found that a perfect 1:1 class ratio wasn't always optimal; for several tasks, an oversampling ratio of 0.5:1 yielded the highest F1-scores, suggesting that over-duplication can introduce its own biases.

Conclusion

The CASIA framework demonstrates that a unified "recipe"—DAP followed by robust fine-tuning—is effective for both classification and NER across multiple languages. Future work will explore incorporating Medical Knowledge Graphs to link social media mentions to structured medical databases like UMLS, providing deeper semantic context.

Summary updated

Abstract

This paper presents a description of our system in SMM4H-2022, where we participated in task 1a,task 4, and task 6 to task 10. There are three main challenges in SMM4H-2022, namely the domain shift problem, the prediction bias due to category imbalance, and the noise in informal text. In this paper, we propose a unified framework for the classification and named entity recognition tasks to solve the challenges, and it can be applied to both English and Spanish scenarios. The results of our system are higher than the median F1-scores for 7 tasks and significantly exceed the F1-scores for 6 tasks. The experimental results demonstrate the effectiveness of our system.
BibTeX
@inproceedings{fu2022casia,
  author = {Jia Fu and Sirui Li and Hui Ming Yuan and Zhucong Li and Zhen Gan and Yubo Chen and Kang Liu and Jun Zhao and Shengping Liu},
  title = {{CASIA@SMM4H'22: A Uniform Health Information Mining System for Multilingual Social Media Texts}},
  booktitle = {SMM4H @ COLING},
  year = {2022},
  url = {https://openreview.net/forum?id=Vk4mn9tJso},
}
Figure