LLM-RM at SemEval-2023 Task 2: Multilingual Complex NER Using XLM-RoBERTa

Rahul Mehta, Vasudeva VarmaACL 2023

TL;DR

This paper tackles the challenging task of multilingual Named Entity Recognition (NER) for complex entities across 12 languages, using a fine-tuned XLM-RoBERTa model. The authors demonstrate the effectiveness of this approach in the SemEval-2023 MultiCoNER II shared task, achieving competitive results despite the inherent difficulties of low-resource languages and intricate entity types.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and categorizing entities like people, organizations, and locations within text. While NER has seen significant advancements, particularly for high-resource languages like English, extending this capability to a multilingual setting, especially for complex and nuanced entities, remains a significant challenge. The SemEval-2023 Task 2, MultiCoNER II, specifically addresses this by focusing on 12 languages, including several low-resource ones, and introducing complex entity types and simulated errors to the test data, making it a more realistic and demanding benchmark. This task aims to push the boundaries of NER systems to handle the diversity and complexity of real-world language data.

Methods

The core of the LLM-RM team's approach is the utilization of the XLM-RoBERTa-base model. XLM-RoBERTa is a powerful, pre-trained transformer model designed for multilingual understanding, having been trained on over 100 languages. The authors fine-tuned this model on the specific datasets provided for each of the 12 languages in the MultiCoNER II task. This fine-tuning process adapts the general language understanding capabilities of XLM-RoBERTa to the specific nuances of NER within each language and the complex entity definitions of the task. For token classification, a linear layer was used as the final output layer. The experimental setup involved training on a single GPU, using a batch size of 4, a learning rate of 2e-5, and the AdamW optimizer with a dropout of 0.2 to prevent overfitting. The model was trained for 15 epochs, with the best performing checkpoint selected based on F1-scores on the development set.

Datasets

The primary dataset used is the MultiCoNER v2 dataset, which was provided for the SemEval-2023 Task 2. This dataset comprises individual language datasets for 12 languages: English, Spanish, Dutch, Russian, Turkish, Portuguese, Korean, Farsi, German, Chinese, Hindi, and Bangla. The dataset is split into training, development, and two test sets: Test-1 (corrupted) and Test-2 (uncorrupted). The paper notes that the uncorrupted test set is significantly larger than the training set for most languages. The task also builds upon the previous year's challenge, MultiCoNER, which introduced complex entities and simulated errors.

Key Insights and Ideas

  • XLM-RoBERTa's Strength in Multilingual NER: The paper reinforces the effectiveness of large, pre-trained multilingual models like XLM-RoBERTa for tackling complex NER tasks across diverse languages, especially when fine-tuned on task-specific data.
  • The Challenge of Complex Entities: The MultiCoNER II task highlights that recognizing syntactically complex entities (e.g., movie titles, creative works) and entities in low-context situations is a significant hurdle for current NER systems.
  • Long-Tail Entity Distribution: The presence of many domain-specific entities with long-tail distributions poses a challenge for creating comprehensive training data, making robust generalization difficult.
  • Performance Drop in Hindi: A notable observation is the significant drop in performance for Hindi from the development set to the test set, suggesting potential domain shift or specific challenges within the Hindi data.

Key Results

The paper reports macro-average precision, recall, and F1-scores for XLM-RoBERTa-base on the development set across the 12 languages. For instance, German achieved an F1-score of 61.62, while Chinese had an F1-score of 25.50, illustrating the varying performance across languages. The authors also analyzed performance by entity type, finding that "Creative Work" entities consistently achieved the highest F1-scores across languages, whereas "Person" and "Product" groups had the lowest. On the final test set, the F1-scores for Hindi, English, German, and Spanish were reported, with Hindi showing the largest performance drop from development to test.

Limitations and Future Work

The authors acknowledge that they were unable to submit results for all languages by the competition deadline, limiting their reported test set scores to Hindi, English, German, and Spanish. For future work, they suggest exploring more recent transformer-based models with larger vocabularies, such as XLM-V, and investigating data augmentation techniques like entity replacement to further improve model performance and robustness.

Summary updated

BibTeX
@inproceedings{mehta2023llm,
  author = {Rahul Mehta and Vasudeva Varma},
  title = {{LLM-RM at SemEval-2023 Task 2: Multilingual Complex NER Using XLM-RoBERTa}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure