Minanto at SemEval-2023 Task 2: Fine-tuning XLM-RoBERTa for Named Entity Recognition on English Data

Antonia Höfer, Mina MottahedinACL 2023

TL;DR

This paper explores fine-tuning XLM-RoBERTa for the challenging task of fine-grained Named Entity Recognition (NER) on English data. The team achieved a respectable F1 score of 51.47%, highlighting that while cross-lingual models offer flexibility, a monolingual approach might be more effective for specific languages like English in this context.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), aiming to identify and classify named entities (like people, organizations, locations) in text. However, recognizing complex named entities, such as media titles, products, or groups, presents a significant challenge. These entities can be ambiguous (e.g., "On the Beach" could be a book title or a phrase) and their taxonomy is constantly evolving. The MultiCoNER shared task specifically targets this by introducing a fine-grained entity taxonomy with over 30 classes and dealing with noisy data, making the task considerably harder. This paper tackles the English portion of this challenge, aiming to improve NER performance on these complex entities.

Methods

The core of the Minanto team's approach involved fine-tuning a pre-trained XLM-RoBERTa-base model. XLM-RoBERTa is a powerful cross-lingual language model, pre-trained on a vast amount of multilingual data, allowing it to learn representations that can be transferred across languages. The process involved:

  1. Data Pre-processing: Standardizing the input data into a format suitable for sequence labeling, where each word is paired with its corresponding entity tag.
  2. Fine-tuning XLM-RoBERTa: Using the Hugging Face library, the XLM-RoBERTa model was fine-tuned on the MultiCoNER English training data. This involved tokenizing the text, feeding the token embeddings into a neural network classifier (a linear layer), and training the model to predict the named entity labels.
  3. Hyperparameter Tuning: The model was trained with a batch size of 16, three epochs, and a learning rate of 3.0 x 10^-5, chosen through empirical experiments.
  4. External Data Integration (Exploratory): The team also experimented with incorporating external datasets (CoNLL 2003++ and WNUT 2016) to potentially boost performance, as this had been successful in previous years. This involved translating the tags from these datasets to match the MultiCoNER taxonomy.

Datasets

  • MultiCoNER Dataset: This is the primary dataset used for the SemEval-2023 Task 2. It includes data from 12 languages, with a fine-grained taxonomy of 33 named entity classes across six categories. The data was collected from three domains: Wikipedia sentences, questions, and search queries. The English portion of this dataset was used for training and evaluation.
  • CoNLL 2003++ and WNUT 2016 Datasets: These were used as external data sources for potential performance enhancement. The paper notes that the tag sets in these datasets differed from MultiCoNER, requiring translation and leading to some loss of information.

Key Insights and Ideas

  • Monolingual vs. Cross-lingual Performance: A significant finding was that for English data, a monolingual model (like RoBERTa-base, though not explicitly detailed in the final submission) might perform better than a cross-lingual model like XLM-RoBERTa. This suggests that while cross-lingual models offer versatility, they might not always be optimal for a specific, well-resourced language.
  • Limited Impact of External Data: Incorporating external datasets, even after tag translation, provided only minor improvements. This was attributed to the differences in granularity between the MultiCoNER dataset and the external ones, and the potential for errors introduced during tag translation.
  • Noise in Test Data: The presence of noisy instances (typing errors) in the test dataset is identified as a factor that likely contributed to lower overall scores compared to previous years.
  • Class Imbalance and Specificity: The performance on individual classes showed a correlation with their frequency in the training set. However, class specificity also played a role, with more specific classes being harder to predict.

Key Results

  • Overall Performance: The Minanto team achieved an overall fine-grained macro-averaged F1 score of 51.47% in the final submission, ranking 28th out of 34 teams.
  • Coarse-grained Performance: The system achieved a coarse-grained F1 score of 70.75%.
  • Best Performing Classes (Fine-grained): HumanSettlement (0.8396), Athlete (0.7466), and Artist (0.7381) were the most accurately predicted classes.
  • Worst Performing Classes (Fine-grained): PrivateCorp (0.0197), Symptom (0.1561), and AerospaceManufacturer (0.3205) were among the least accurately predicted.
  • Best Performing Classes (Coarse-grained): PER (0.9019) was the best performing class, while PROD (0.5143) was the worst.
  • External Data Impact: While external data initially reduced the F1 score on the development set, it led to a slight increase of 1.07% on the test data, but its impact was deemed "very limited."

Limitations and Future Work

  • Monolingual Model Performance: The paper notes that after submission, they realized a RoBERTa-base model (English-only) might have performed better than XLM-RoBERTa-base. This suggests future work could focus on optimizing monolingual models for specific languages.
  • External Data Standardization: The authors highlight that standardized fine-grained labels across external datasets would greatly facilitate future work and improve the integration of external knowledge.
  • Noisy Data Handling: The presence of noisy data in both the training and test sets is a limitation. Future work could explore more robust methods for handling noisy inputs.
  • Model Architecture: While XLM-RoBERTa is powerful, exploring other architectures or ensemble methods could potentially yield better results. The paper also mentions that the impact of external data was limited, suggesting that more sophisticated methods for integrating such data might be needed.

Summary updated

BibTeX
@inproceedings{hofer2023minanto,
  author = {Antonia Höfer and Mina Mottahedin},
  title = {{Minanto at SemEval-2023 Task 2: Fine-tuning XLM-RoBERTa for Named Entity Recognition on English Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure