Abstract
This paper describes the system developed for the Social Media Mining for Health (SMM4H) 2022 SocialDisNER task. We utilized various pre-trained language models trained on Spanish biomedical literature and Spanish tweets. Our research highlights the significant impact of tokenization quality and the effects of incorporating silver standard annotations during training. The final model achieved a strict F1 score of 80.3% on the test set, representing a +12.8% improvement over the average results of all challenge submissions.
Introduction
Public health surveillance increasingly leverages social media to monitor disease outbreaks and patient experiences in real-time. However, extracting structured medical information from platforms like Twitter is challenging due to the informal nature of the text, which includes misspellings, slang, abbreviations, and hashtags. These difficulties are compounded in non-English contexts where NLP tools may be less mature. The SocialDisNER task specifically focuses on identifying disease mentions in Spanish tweets, requiring systems to navigate both linguistic nuances and the "noisy" nature of social media data.
Methodology
Our approach centered on a transformer-based architecture, specifically evaluating several BERT-type models pre-trained on Spanish corpora.
Model Selection
We benchmarked five pre-trained models to identify the most effective base for our NER task. These included models trained on general Spanish text and those specialized in the biomedical domain. As shown in Figure 1, the model by Carrino et al. (2022), specifically the PlanTL-GOB-ES/bsc-bio-es model, demonstrated superior performance in initial tests.
Custom Tokenization and Preprocessing
To address the irregularities of tweet data, we implemented a specialized tokenization strategy. Standard tokenizers often struggle with hashtags and punctuation attached to words. Our "Improved" strategy involved:
- Splitting text by spaces and isolating punctuation.
- Separating hashtags (#) and mentions (@) from the following text.
- Replacing emojis with a placeholder character (
@) to preserve character offsets for evaluation without introducing semantic noise.
Training and Data
The system was trained using the Gold Standard (GS) dataset of 7,500 expert-annotated tweets. We also experimented with a Silver Standard dataset of 85,077 machine-labeled tweets to determine if increased data volume could compensate for lower label quality.
Results and Discussion
The performance of our system was evaluated using strict and overlap F1 measures. The results, summarized in Figure 2, reveal critical insights into model optimization.
The Impact of Tokenization
The transition from "Simple" to "Improved" tokenization yielded the most substantial performance gain. On the validation set, the strict F1 score rose from 71.9% to 82.9%, a 6.1% absolute increase. This underscores that for social media NER, preprocessing is as critical as model architecture.
Silver Standard Data
Contrary to expectations, adding the large-scale Silver Standard dataset significantly degraded performance. The strict F1 score dropped from 76.8% to 62.4% when silver data was included. This suggests that the noise inherent in machine-generated labels can be detrimental for specialized tasks like medical entity recognition, where precision is paramount.
Final Evaluation
On the official test set, our model achieved a strict F1 score of 80.3%. This result significantly outperformed the average (67.5%) and median (76.1%) scores of all participants in the SocialDisNER challenge, demonstrating the robustness of our domain-specific model and optimized preprocessing pipeline.
Conclusion
Our participation in the SocialDisNER task highlights the importance of domain-specific pre-training and meticulous text preprocessing for mining health data from social media. While our system achieved state-of-the-art results, future work will explore ensemble methods to better capture the intersection of medical terminology and social media vernacular, as well as more sophisticated post-processing to resolve boundary errors in entity detection.
Summary updated