Abstract
Adrian Garcia Hernandez, Leung Wai Liu, Akshat Gupta, Vineeth Ravi, Saheed O. Obitayo, Xiaomo Liu, Sameena Shah. Proceedings of The Seventh Workshop on Social Media Mining for Health Applications, Workshop & Shared Task.
2022.
Introduction
While Natural Language Processing (NLP) has advanced rapidly, the field remains heavily skewed toward English. This "language gap" is particularly problematic in public health surveillance, where monitoring global trends—like the spread of COVID-19 symptoms—requires analyzing social media data in multiple languages.
This paper addresses Task 5 of the Social Media Mining for Health Applications (SMM4H) 2022 competition. The goal is a three-way classification of Spanish tweets into self-reports, non-personal reports, and literal news mentions. The primary challenge lies in the nuance of Spanish phrasing and the significant class imbalance in the data, where news mentions far outnumber personal reports.
Methodology
The research team developed a robust system based on Multiple-Model Ensembling. Rather than relying on a single architecture, they leveraged the collective intelligence of several state-of-the-art models:
- BERT-base-multilingual: A standard for cross-lingual tasks.
- BETO (Spanish-BERT): A model specifically pre-trained on a massive Spanish corpus to capture local linguistic nuances.
- XLM-RoBERTa: A powerful cross-lingual model known for its performance on diverse datasets.
Handling Class Imbalance
To address the dominance of news mentions in the training set, the authors experimented with three specialized data-balancing strategies: Equalized (over-sampling minority classes), Over-Under (a hybrid approach), and Reversed (drastically over-sampling the "Self-reports" class).
The final predictions were generated using a majority-vote ensemble. The team compared two different data-splitting strategies—an 80/20 split and a 50/50 split—to evaluate generalization on unseen data.
Datasets
The study utilized the official SMM4H 2022 Task 5 dataset. As shown in Figure 1, the training set of 10,051 labeled tweets is heavily skewed toward Lit-News_Mentions, which accounts for approximately 60% of the data.
The validation set contained 3,577 labeled tweets, while the test set consisted of 6,850 unlabeled tweets used for the final competition ranking. The unknown distribution of the test set necessitated experimentation with different internal validation splits.
Key Insights and Results
The ensemble approach proved highly effective, demonstrating a distinct trade-off between class-specific optimization and overall accuracy. The 80/20 split ensemble was better at identifying the positive Self_reports class, while the 50/50 split provided a better micro-averaged F1-score.
As illustrated in Figure 3, the team's models achieved a micro-averaged F1-score of 0.85 (for the 50/50 split) and 0.84 (for the 80/20 split). Both scores were reported as higher than the competition's median result of 0.84. Furthermore, the Spanish-specific BETO model consistently performed well, proving that language-specific pre-training remains a high-value asset. The ensemble method also provided stability, reducing the risk of individual model variance by averaging the votes of 25 different iterations.
Conclusion and Future Work
The researchers identified several areas for future improvement, including noise reduction (removing handles and hyperlinks) and data augmentation via back-translation to further combat class imbalance. Despite these limitations, the ensemble-based system successfully demonstrated how combining multiple transformer models can mitigate the challenges of non-English NLP in public health contexts.
Summary updated