NamedEntityRangers at SemEval-2022 Task 11: Transformer-based Approaches for Multilingual Complex Named Entity Recognition

Amina Miftahova, Alexander Pugachev, Artem Skiba, Ekaterina Artemova, Tatiana Batura, Pavel Braslavski, Vladimir IvanovNAACL 2022

TL;DR

This paper explores two transformer-based approaches for multilingual complex named entity recognition (NER) on the challenging MultiCoNER dataset. Surprisingly, a traditional token classification method using RemBERT outperformed a more recent template-free approach with mT5, suggesting that simpler methods can still be highly effective for complex NER tasks.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and categorizing named entities (like people, organizations, and locations) in text. However, real-world text often presents challenges beyond simple entity types. These include:

  • Multilingualism: Handling text in multiple languages, especially those from different language families and using different scripts.
  • Complex Entities: Recognizing entities that are syntactically intricate, ambiguous, or occur infrequently (the "long tail"). Examples include titles of creative works, products, or even country names that can be ambiguous.
  • Noisy and Non-uniform Data: Dealing with datasets that are not perfectly clean or have an uneven distribution of entity types.

The SemEval-2022 Task 11 (MultiCoNER) specifically targets these challenges by providing a dataset with 11 languages and complex entity types, aiming to push the boundaries of multilingual NER systems. This paper investigates how state-of-the-art transformer models perform on this demanding task and compares two distinct approaches.

Methods

The authors implemented and evaluated two main approaches, both leveraging pre-trained multilingual transformer models:

  1. Token Classification: This is a mainstream NER technique where each token in the input sequence is assigned a specific tag (e.g., "B-PER" for the beginning of a person entity, "I-LOC" for inside a location entity, or "O" for outside any entity). They used RemBERT, a multilingual BERT-based model, fine-tuned with a linear layer on top of its hidden states. They also explored ensembling three RemBERT models trained with different random seeds, using hard or soft voting for final predictions.
  2. Template-Free Approach: This more recent paradigm, inspired by models like T5, treats NER as a text-to-text generation task. The model is trained to "translate" the input sequence by replacing named entity spans with predefined label words. For instance, an entity like "New York City" might be replaced by a generic label like "LOC". They used mT5-Large and mT5-XL, multilingual variants of the T5 model, for this approach.

Datasets

The primary dataset used is the MultiCoNER dataset, introduced for SemEval-2022 Task 11. This dataset is notable for its:

  • Multilingualism: Covering 11 languages from different families and scripts.
  • Complex Entity Types: Including "Group," "Location," "Person," "Creative Work," and "Product."
  • Challenging Nature: Designed to be more realistic with ambiguous and long-tail entities.

The authors also mention using datasets like CoNLL03 and MIT-Movie for preliminary experiments with the template-free approach, which are standard benchmarks for NER.

Key Insights and Ideas

  • Token Classification Reigns Supreme (for now): The most striking finding is that the traditional token classification approach with RemBERT significantly outperformed the template-free mT5 approach on the MultiCoNER task. This suggests that despite the advancements in generative models, well-established methods can still be highly competitive, especially when dealing with complex entity structures and potential dataset inconsistencies.
  • Dataset Inconsistencies Hurt Generative Models More: The authors observed that the template-free approach was more sensitive to inconsistencies in the MultiCoNER dataset, particularly regarding "mislabeled-as-O" entities (where an entity is incorrectly labeled as "O") and punctuation handling. This implies that generative models, which rely on precise reconstruction, are more vulnerable to noisy labels than token classifiers.
  • Entity Length Matters: Analysis revealed that RemBERT (token classification) performed better on shorter entities, while mT5-XL (template-free) showed an advantage on longer entities. This difference, combined with the shorter average sentence length in the test set compared to the training/dev sets, might explain the overall performance gap.
  • Ensembling Offers Limited Gains: While ensembling multiple RemBERT models was explored, it did not lead to significant improvements, suggesting that individual models were already converging to similar, high-quality predictions.

Key Results

  • Performance Gap: The Token Classification approach achieved an F1 score of 0.81 on the test set, while the Template-Free approach (mT5-XL) achieved 0.54. Both results were within 10% of the top-performing solutions for the multilingual NER task.
  • Error Analysis: For both models, errors were most common for "Product" and "Creative Work" entities, which are among the complex types. The token classification model was more prone to predicting "O" (non-entity) tags, while the template-free model struggled more with reconstruction due to dataset issues.
  • Language Performance: The fine-tuned RemBERT model performed best on English, Dutch, and German, and worst on Farsi and Russian.

Limitations and Future Work

  • Dataset Issues: The authors highlight significant dataset issues, including inconsistent labeling of entities (especially "mislabeled-as-O" entities) and punctuation handling. These inconsistencies likely impacted the performance of both models, particularly the template-free approach. They suggest that clearer definitions and more consistent labeling are crucial for future work.
  • Template-Free Model Sensitivity: The template-free approach's difficulty with reconstructing spans unambiguously and handling punctuation issues points to areas for improvement in the methodology or preprocessing.
  • Ensemble Performance: The limited benefit from ensembling warrants further investigation.
  • Future Directions: The authors suggest that addressing dataset issues and further exploring the strengths of different approaches for various entity lengths and types would be valuable. They also note that evaluating systems for multilingual NER with complex entities remains a challenging area.

Summary updated

BibTeX
@inproceedings{miftahova2022namedentityrangers,
  author = {Amina Miftahova and Alexander Pugachev and Artem Skiba and Ekaterina Artemova and Tatiana Batura and Pavel Braslavski and Vladimir Ivanov},
  title = {{NamedEntityRangers at SemEval-2022 Task 11: Transformer-based Approaches for Multilingual Complex Named Entity Recognition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure