Identification of profession & occupation in Health-related Social Media using tweets in Spanish

Victoria Pachón, Jacinto Mata Vázquez, Juan Luís Domínguez OlmedoNAACL 2021

TL;DR

This paper tackles the challenge of identifying professions and occupations in Spanish-language health-related tweets. The authors demonstrate that fine-tuning Spanish-specific BERT models, like BETO, significantly outperforms models relying on English translations, achieving a high F1-score of 0.92 for positive class identification.

What Problem Does This Paper Address?

The COVID-19 pandemic highlighted the critical need to understand the impact of the virus on different professions, particularly those on the front lines. Identifying these vulnerable occupations is crucial for implementing targeted preventive measures. However, a significant hurdle exists in processing and analyzing social media data in languages other than English, especially for tasks like named entity recognition (NER) or classification. This paper specifically addresses the challenge of identifying profession and occupation mentions within Spanish tweets, a task that is often hindered by a lack of readily available, high-quality resources for Spanish NLP. The authors hypothesize that directly using Spanish-pretrained models will yield better results than relying on translated data.

Methods

The core of the methodology involves fine-tuning pre-trained transformer models for a binary classification task: determining if a tweet mentions an occupation. The authors compare several approaches:

  • Baseline: A Bidirectional Long-Short Term Memory (BiLSTM) RNN with a skipgram embedding layer.
  • Transformer Models:
    • BETO: A Spanish-specific BERT model.
  • mBERT: A multilingual BERT model.
  • RuPERTa: A Spanish RoBERTa model.
  • English BERT (cased and uncased): Used with translated datasets for comparison.

The key innovation lies in directly using Spanish-pretrained models (BETO, RuPERTa) and comparing them against multilingual models and English models with translated data. This approach aims to leverage the nuances of the Spanish language more effectively.

Datasets

The primary dataset used is from the ProfNer-ST: Task 7a (Tweet binary classification) challenge. The authors also mention using the training and test datasets provided by the organizers. For experiments, they utilized three versions of the dataset:

  • Original: The raw tweet text.
  • URLs_removed: Tweets with URLs stripped.
  • Hashtags_URLs_removed: Tweets with both URLs and hashtags removed.

The training dataset was split to create a validation set (30%).

Key Insights and Ideas

  • Spanish-first is best: Directly using models pre-trained on Spanish corpora (like BETO) is significantly more effective than translating Spanish text to English and then using English-pretrained models. This underscores the importance of language-specific resources for NLP tasks.
  • Transformer dominance: Transformer-based models, particularly BETO, outperform classical machine learning approaches like BiLSTM for this task.
  • Preprocessing impact: While extensive preprocessing wasn't deemed necessary for tweets, removing URLs and hashtags had a minor impact on performance, suggesting that these elements don't significantly hinder the model's ability to identify occupations in this context.

Key Results

The paper reports impressive results, with the fine-tuned BETO-cased model achieving an F1-score of 0.92 for the positive class on the evaluation dataset. This was the best performance among all tested models and configurations. The BETO models (both cased and uncased) consistently outperformed mBERT, RuPERTa, and the English BERT models across various preprocessing scenarios.

Limitations and Future Work

The authors acknowledge that while their approach proved effective, they want to continue testing BETO in other contests. Future work could involve exploring more advanced preprocessing techniques for tweets or investigating other Spanish-specific transformer architectures. The paper also implicitly points to the ongoing need for more robust NLP resources for under-resourced languages like Spanish.

Summary updated

BibTeX
@inproceedings{pachon2021identification,
  author = {Victoria Pachón and Jacinto Mata Vázquez and Juan Luís Domínguez Olmedo},
  title = {{Identification of profession \& occupation in Health-related Social Media using tweets in Spanish}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure