SAB at SemEval-2023 Task 2: Does Linguistic Information Aid in Named Entity Recognition?

Siena BialesACL 2023

TL;DR

This paper investigates whether incorporating linguistic information like part-of-speech (POS) and dependency relations can improve Named Entity Recognition (NER) performance, especially for complex entities across multiple languages. The study found that while multi-task learning with these linguistic features didn't boost performance and even slightly degraded it, the Hindi track showed surprisingly strong results due to a lower proportion of unseen entities.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that aims to identify and classify named entities (like people, organizations, or locations) in text. While traditional NER focused on simpler entities, recent efforts, like the SemEval-2023 Task 2 (MultiCoNER II), are tackling "complex named entities" (e.g., creative works, products) which are more challenging due to their varied forms and the ever-expanding nature of their domains. Historically, NER models have benefited from external knowledge bases or gazetteers. However, the impact of integrating linguistic information (like grammatical roles and sentence structure) directly into the model, particularly in a multilingual setting, remains less explored. This paper aims to fill that gap by examining whether adding POS and dependency relation information to a powerful transformer model can enhance its ability to recognize these complex entities.

Methods

The researchers employed a multi-task learning approach using XLM-RoBERTa (XLM-R) as the base model. XLM-R is a robust, multilingual transformer model pre-trained on a massive dataset. The core idea was to train a single XLM-R model with separate "heads" for three tasks: Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and Dependency Relation parsing. This setup allows the model to learn from all three tasks simultaneously, potentially leveraging shared representations.

For preprocessing, the original CoNLL-formatted data was converted to JSON. POS tags were obtained using the Stanza toolkit for most languages, with a specific Python library used for Bangla. Dependency relation labels were also extracted. The model was then trained with a shared XLM-R encoder and task-specific decoders for NER, POS, and dependency parsing. Hyperparameter tuning was primarily conducted on German and English datasets due to their different training sample sizes, and the best settings were then applied to other languages.

Datasets

The primary dataset used was MultiCoNER v2, the second iteration of the MultiCoNER dataset. This dataset is designed to address the challenges of complex named entity recognition across 12 languages. It features 33 fine-grained entity classes, which can be generalized into 6 coarse-grained categories: location (LOC), creative work (CW), group (GRP), person (PER), product (PROD), and medical (MED). The dataset includes simulated errors in some test sets to mimic real-world scenarios.

Key Insights and Ideas

  • Linguistic Features Don't Always Help: Contrary to expectations, integrating POS and dependency relation information into the multi-task setup did not improve NER performance and, in some cases, slightly degraded it. This suggests that powerful pre-trained models like XLM-R might already implicitly capture much of this linguistic information.
  • Hindi's Unexpected Success: The Hindi language track yielded the best results for the team. This is attributed to a significantly lower percentage of "unseen entities" in the Hindi test set compared to English. This highlights the crucial role of data distribution and familiarity in model performance.
  • Noise Sensitivity: The models showed a noticeable drop in performance (around 10 percentage points) on the noisy test data, indicating a lack of robustness to errors.
  • Complex Entity Challenges: The results align with expectations, showing that identifying "open-class" entity types like products and creative works is generally harder than more established categories like locations and persons.

Key Results

The paper reports macro-averaged F1 scores for each language track. While the multi-task model generally underperformed the baseline NER model, the best performance was achieved on the Hindi track. For example, the overall F1 score for Hindi was 0.62, while for English it was 0.51. The study also noted that the F1 scores on noisy test data dropped by approximately 10 percentage points compared to clean data across affected language tracks.

Limitations and Future Work

The authors acknowledge several limitations. The multi-task training span of 10 epochs might have been too short for the model to effectively learn from the additional linguistic tasks. They also speculate that the quality of the POS and dependency relation labels generated by Stanza might not be perfect and could be introducing noise.

Future work could involve:

  • Extending Training Epochs: Investigating if longer training periods would allow the model to better leverage the linguistic information.
  • Altering Learning Rates: Experimenting with different learning rates for the NER task to see if it can be prioritized.
  • Exploring Alternative Taggers: Using different tools for POS and dependency relation extraction to assess the impact of tagger quality.
  • Tokenizer Variations: Testing character-based tokenizers instead of sub-word tokenizers to see their effect on robustness to simulated errors.
  • Separating Task Losses: Investigating if separating the loss functions for each task in the multi-task setup would yield different results.

Summary updated

BibTeX
@inproceedings{biales2023sab,
  author = {Siena Biales},
  title = {{SAB at SemEval-2023 Task 2: Does Linguistic Information Aid in Named Entity Recognition?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure