Lasige-BioTM at ProfNER: BiLSTM-CRF and contextual Spanish embeddings for Named Entity Recognition and Tweet Binary Classification

Pedro Ruas, Vitor Andrade, Francisco CoutoNAACL 2021

TL;DR

This paper presents the Lasige-BioTM team's approach to the ProfNER shared task, focusing on identifying professions and occupations in Spanish social media. They achieved strong results using a BiLSTM-CRF model with contextual Spanish embeddings for Named Entity Recognition (NER) and a rule-based system for tweet classification, outperforming the median in one sub-track.

What Problem Does This Paper Address?

The paper tackles the challenge of accurately identifying and classifying mentions of professions and occupations within Spanish tweets. This is crucial for understanding employment trends, public health discussions, and social dynamics reflected in online discourse. The ProfNER shared task specifically aimed to advance research in this area by providing a dedicated corpus and evaluation framework for Spanish social media data. The task was divided into two sub-tracks: binary classification of tweets (does it contain an occupation mention?) and detailed Named Entity Recognition (NER) for these mentions.

Methods

The Lasige-BioTM team employed two primary methods:

  • Named Entity Recognition (NER): For sub-track B, they utilized a BiLSTM-CRF (Bidirectional Long Short-Term Memory with Conditional Random Field) model. This is a standard and powerful architecture for sequence labeling tasks like NER. They experimented with different combinations of pre-trained embeddings, including:
  • "Base" embeddings: A combination of FLAIR contextual embeddings (trained on general Spanish text) and Spanish FastText embeddings.
  • "Twitter" embeddings: FastText embeddings specifically trained on Spanish COVID-19 tweets.
  • "Medium" embeddings: A blend of FLAIR, general Spanish FastText, and Twitter-specific FastText embeddings. They also explored data augmentation techniques to improve the training data.

  • Tweet Binary Classification: For sub-track A, they developed a rule-based module. This module leveraged the output of their NER system (from sub-track B). If the NER system identified at least one occupation-related entity in a tweet, the tweet was classified as positive (label "1"); otherwise, it was classified as negative (label "0").

Datasets

The primary dataset used was the ProfNER corpus. This corpus consists of 10,000 Spanish tweets annotated with entities related to professions, employment statuses, and work activities. The annotations fall into four categories: "PROFESION" (profession), "SITUACION_LABORAL" (employment status), "ACTIVIDAD" (activity), and "FIGURATIVA" (figurative). For evaluation, sub-track A used all categories, while sub-track B focused only on "PROFESION" and "SITUACION_LABORAL". The authors also mention using a "gazetteer" (a list of known entities) for occupations provided by an organization.

Key Insights and Ideas

  • Contextual Embeddings Matter: The "Base" model, which combined general contextual embeddings (FLAIR) with word embeddings (FastText), outperformed the "Twitter" model that used only Twitter-specific embeddings. This suggests that broader contextual understanding can be more beneficial than highly specialized but potentially less diverse embeddings, at least in this specific scenario.
  • Data Augmentation's Mixed Impact: While data augmentation was explored, the models trained on augmented data generally showed lower performance compared to those trained solely on the original training data. This highlights the importance of the quality and nature of augmentation techniques.
  • Rule-Based for Binary Classification is Effective: A simple rule-based approach, directly using NER results, proved effective for the binary tweet classification task, demonstrating that complex models aren't always necessary for simpler classification problems when good upstream information is available.
  • Twitter Data Nuances: The error analysis revealed that general embeddings sometimes struggle with Twitter-specific nuances, such as distinguishing between a professional mention and a Twitter handle (e.g., "@MorenaSenadores"). This indicates a need for careful consideration of domain-specific language.

Key Results

  • Sub-track A (Tweet Binary Classification): The Lasige-BioTM team achieved a F1-score of 0.917, which was 0.031 points higher than the median of competing teams.
  • Sub-track B (NER): The team achieved a F1-score of 0.727, which was 0.034 points lower than the median.
  • The "Base" model (BiLSTM-CRF with general contextual and word embeddings) trained on the original training file achieved the best performance in the practice phase for both sub-tracks, with F1-scores of 0.908 (sub-track A) and 0.716 (sub-track B).

Limitations and Future Work

  • Limited Time for Training: The authors acknowledge that they had limited time for training, which restricted their ability to perform extensive hyper-parameter optimization for the BiLSTM-CRF model (e.g., learning rate, hidden size, epochs).
  • Exploring More Embeddings: They plan to investigate other contextualized embeddings, as these have shown promise in achieving better performance than classical word embeddings.
  • Named Entity Linking (NEL): For improved tweet classification, they intend to explore Named Entity Linking tools. By linking recognized entities to structured vocabularies (like MeSH or DBpedia), they aim to leverage hierarchical relationships to better determine if an entity truly represents an occupation.
  • Hyper-parameter Optimization: Future work will focus on optimizing hyperparameters for the BiLSTM-CRF model.

Summary updated

BibTeX
@inproceedings{ruas2021lasige,
  author = {Pedro Ruas and Vitor Andrade and Francisco Couto},
  title = {{Lasige-BioTM at ProfNER: BiLSTM-CRF and contextual Spanish embeddings for Named Entity Recognition and Tweet Binary Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure