The ProfNER shared task on automatic recognition of occupation mentions in social media: systems, evaluation, guidelines, embeddings and corpora

Antonio Miranda-Escalada, Eulàlia Farré-Maduell, Salvador Lima-López, Luis Gascó, Vicent Briva-Iglesias, Marvin Agüero-Torales, Martin KrallingerNAACL 2021

TL;DR

This paper introduces ProfNER, the first shared task and benchmark for automatically recognizing occupation mentions in Spanish social media, specifically focusing on COVID-19 related contexts. They provide a high-quality annotated corpus, guidelines, and embeddings, enabling advancements in understanding at-risk occupational groups and informing public health policies.

What Problem Does This Paper Address?

Identifying occupations in text is crucial for various applications like competitive intelligence, demographic analysis, and public health monitoring. However, existing efforts in this area have been limited, particularly for social media data and in languages other than English. This is largely due to a lack of standardized annotation guidelines and high-quality, publicly available datasets. The COVID-19 pandemic highlighted the urgent need to monitor at-risk occupational groups, especially those in direct contact with infectious agents or experiencing mental health strain. To address this gap, the authors organized the ProfNER shared task to spur the development of NLP tools for occupation recognition in Spanish social media.

Methods

The ProfNER shared task was structured into two subtracks to handle the inherent class imbalance in social media data:

  • Subtrack A: Tweet Binary Classification: This task involved classifying tweets into two categories: those that mention occupations and those that do not.
  • Subtrack B: NER Offset Detection and Classification: This subtrack focused on identifying the exact text spans that refer to occupations and classifying them into predefined categories.

The task utilized a manually annotated corpus of 10,000 COVID-related Spanish tweets. Participants were provided with annotation guidelines, an occupation gazetteer, and pre-trained FastText embeddings. Systems were evaluated using F1-scores for Subtrack A and micro-averaged F1-scores for Subtrack B. The best-performing systems leveraged transformer architectures and contextualized embeddings with BiLSTM-CRF models.

Datasets

The primary dataset introduced is the ProfNER Gold Standard Corpus, consisting of 10,000 COVID-related tweets in Spanish, manually annotated with four types of occupation mentions: PROFESION (profession), ACTIVIDAD (activity), SITUACION LABORAL (working status), and FIGURATIVA (figurative use of occupations). This corpus was split into training (60%), development (20%), and test (20%) sets. An additional 25,000 tweets were provided as a background set without annotations, which participants' predictions formed the ProfNER Silver Standard Corpus. The paper also mentions the release of a machine-translated English version of the corpus and FastText embeddings trained on 140 million COVID-related Spanish Twitter posts.

Key Insights and Ideas

  • The Importance of Language-Specific Resources: The ProfNER task highlights the critical need for language-specific resources (corpora, guidelines, embeddings) for effective NLP in non-English languages, especially for specialized domains like health and occupations.
  • Handling Class Imbalance is Crucial: The two-subtrack approach (tweet classification followed by NER) effectively addresses the significant class imbalance in social media data, where occupation mentions are relatively rare.
  • Transformer Architectures Dominate: The success of the top-performing systems, particularly those based on transformer architectures, underscores their current state-of-the-art performance in NLP tasks.
  • SITUATION LABORAL is a Challenge: The analysis revealed that mentions of "SITUACION LABORAL" (working status) were less frequently detected by participants compared to other occupation types, indicating a specific challenge in this category.
  • Punctuation and Special Characters are Difficult: Mentions containing punctuation (like # or @) and capital letters were often missed by the systems, suggesting a need for more robust handling of noisy social media text.

Key Results

  • Participation: 35 teams registered for the task, with 11 submitting a total of 27 runs.
  • Tweet Classification (Subtrack A): The best systems achieved an F1-score of 0.93.
  • Named Entity Recognition (Subtrack B): The best-performing system, developed by Recognai (an industry participant), achieved an F1-score of 0.839. This system utilized a transformer architecture. The second-best system from MIC-NLP achieved an F1-score of 0.824 using BiLSTM-CRF with contextualized embeddings.
  • Annotation Quality: The manually annotated corpus achieved a high Inter-Annotator Agreement (pairwise) of 0.919, indicating good quality and consistency.

Limitations and Future Work

  • Limitations:
    • The paper notes that even with correct boundary detection, entity length did not seem to influence predictability in their corpus.
  • The analysis of results indicated that mentions with punctuation and capital letters were rarely predicted.
  • The "SITUACION LABORAL" category proved more challenging for participants.
  • Future Work:
    • The authors suggest that the ProfNER ecosystem (corpus, systems, guidelines, embeddings) can serve as a template for future shared tasks and for building systems that enable subpopulation characterization in social media.
  • They emphasize the potential of these resources to assist public health policymakers in preventing and managing current and future epidemics.
  • The paper also calls for shared tasks and community benchmarks that incorporate end-user involvement and consider the robustness and implementation of NLP components, especially given the dynamic nature of social media data.

Summary updated

BibTeX
@inproceedings{mirandaescalada2021profner,
  author = {Antonio Miranda-Escalada and Eulàlia Farré-Maduell and Salvador Lima-López and Luis Gascó and Vicent Briva-Iglesias and Marvin Agüero-Torales and Martin Krallinger},
  title = {{The ProfNER shared task on automatic recognition of occupation mentions in social media: systems, evaluation, guidelines, embeddings and corpora}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure