KU_ED at SocialDisNER: Extracting Disease Mentions in Tweets Written in Spanish

Antoine Lain, Wonjin Yoon, Hyunjae Kim, Jaewoo Kang, Ian SimpsonSMM4H @ COLING 2022 · OpenReview

Abstract

This paper describes the system developed for the Social Media Mining for Health (SMM4H) 2022 SocialDisNER task. We utilized various pre-trained language models trained on Spanish biomedical literature and Spanish tweets. Our research highlights the significant impact of tokenization quality and the effects of incorporating silver standard annotations during training. The final model achieved a strict F1 score of 80.3% on the test set, representing a +12.8% improvement over the average results of all challenge submissions.

Introduction

Public health surveillance increasingly leverages social media to monitor disease outbreaks and patient experiences in real-time. However, extracting structured medical information from platforms like Twitter is challenging due to the informal nature of the text, which includes misspellings, slang, abbreviations, and hashtags. These difficulties are compounded in non-English contexts where NLP tools may be less mature. The SocialDisNER task specifically focuses on identifying disease mentions in Spanish tweets, requiring systems to navigate both linguistic nuances and the "noisy" nature of social media data.

Methodology

Our approach centered on a transformer-based architecture, specifically evaluating several BERT-type models pre-trained on Spanish corpora.

Model Selection

We benchmarked five pre-trained models to identify the most effective base for our NER task. These included models trained on general Spanish text and those specialized in the biomedical domain. As shown in Figure 1, the model by Carrino et al. (2022), specifically the PlanTL-GOB-ES/bsc-bio-es model, demonstrated superior performance in initial tests.

Comparison of pre-trained models for disease mention extraction.

Custom Tokenization and Preprocessing

To address the irregularities of tweet data, we implemented a specialized tokenization strategy. Standard tokenizers often struggle with hashtags and punctuation attached to words. Our "Improved" strategy involved:

  • Splitting text by spaces and isolating punctuation.
  • Separating hashtags (#) and mentions (@) from the following text.
  • Replacing emojis with a placeholder character (@) to preserve character offsets for evaluation without introducing semantic noise.

Training and Data

The system was trained using the Gold Standard (GS) dataset of 7,500 expert-annotated tweets. We also experimented with a Silver Standard dataset of 85,077 machine-labeled tweets to determine if increased data volume could compensate for lower label quality.

Results and Discussion

The performance of our system was evaluated using strict and overlap F1 measures. The results, summarized in Figure 2, reveal critical insights into model optimization.

Performance comparison of different model configurations and test set results.

The Impact of Tokenization

The transition from "Simple" to "Improved" tokenization yielded the most substantial performance gain. On the validation set, the strict F1 score rose from 71.9% to 82.9%, a 6.1% absolute increase. This underscores that for social media NER, preprocessing is as critical as model architecture.

Silver Standard Data

Contrary to expectations, adding the large-scale Silver Standard dataset significantly degraded performance. The strict F1 score dropped from 76.8% to 62.4% when silver data was included. This suggests that the noise inherent in machine-generated labels can be detrimental for specialized tasks like medical entity recognition, where precision is paramount.

Final Evaluation

On the official test set, our model achieved a strict F1 score of 80.3%. This result significantly outperformed the average (67.5%) and median (76.1%) scores of all participants in the SocialDisNER challenge, demonstrating the robustness of our domain-specific model and optimized preprocessing pipeline.

Conclusion

Our participation in the SocialDisNER task highlights the importance of domain-specific pre-training and meticulous text preprocessing for mining health data from social media. While our system achieved state-of-the-art results, future work will explore ensemble methods to better capture the intersection of medical terminology and social media vernacular, as well as more sophisticated post-processing to resolve boundary errors in entity detection.

Summary updated

Abstract

This paper describes our system developed for the Social Media Mining for Health (SMM4H) 2022 SocialDisNER task. We used several types of pre-trained language models, which are trained on Spanish biomedical literature or Spanish Tweets. We showed the difference in performance depending on the quality of the tokenization as well as introducing silver standard annotations when training the model. Our model obtained a strict F1 of 80.3% on the test set, which is an improvement of +12.8% F1 (24.6 std) over the average results across all submissions to the SocialDisNER challenge.
BibTeX
@inproceedings{lain2022ku,
  author = {Antoine Lain and Wonjin Yoon and Hyunjae Kim and Jaewoo Kang and Ian Simpson},
  title = {{KU\_ED at SocialDisNER: Extracting Disease Mentions in Tweets Written in Spanish}},
  booktitle = {SMM4H @ COLING},
  year = {2022},
  url = {https://openreview.net/forum?id=9VfNvGtg2I},
}
Figure