DS4DH at SemEval-2022 Task 11: Multilingual Named Entity Recognition Using an Ensemble of Transformer-based Language Models

Hossein Rouhizadeh, Douglas TeodoroNAACL 2022

TL;DR

This paper tackles the challenge of identifying and classifying named entities across 11 languages in short, complex texts, a task crucial for many NLP applications. The authors propose an ensemble method combining multiple transformer-based language models, achieving a respectable F1 score of 65.2% and placing 20th in the SemEval-2022 competition.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental NLP task, but existing models often struggle with real-world data. These challenges include dealing with "long-tail" entities (rarely seen), emerging entities (newly appearing), and linguistically complex entities (like gerunds or full clauses). Furthermore, many NER systems are trained on well-formed news text with ample context, which doesn't translate well to shorter, noisier texts. A significant gap also exists in NER research for languages other than English, particularly low-resource languages. SemEval-2022 Task 11, "Multilingual Complex Named Entity Recognition (MultiCoNER)," directly addresses these issues by focusing on short, low-contextual texts across 11 diverse languages.

Methods

The core of the proposed method is an ensemble strategy built upon pre-trained transformer-based language models. The authors first fine-tuned several multilingual transformer models, including Multilingual-BERT, XLM-ROBERTa (base and large), and DistilBERT-Multilingual, on the MultiCoNER training dataset. For each fine-tuned model, they experimented with adding either a fully connected neural network (FCNN) layer or a conditional random fields (CRF) layer on top of the transformer architecture. To handle sub-tokenization in transformer models, they assigned the label of the first sub-token to the entire word.

The ensemble approach involved selecting the top three best-performing models (based on development set performance) and using a majority vote to determine the final entity label. If a tie occurred among the three models, the label predicted by the best-performing individual model was chosen.

Datasets

The primary dataset used for training and evaluation was the multilingual dataset provided for SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER). This dataset comprises entity-annotated sentences in 11 languages: English, Spanish, Dutch, Russian, Turkish, Korean, Farsi, German, Chinese, Hindi, and Bangla. The dataset includes six entity types: Person, Location, Production, Corporation, Group, and Creative Work. The authors also mention dividing the official training set into 80% for training and 20% for development to tune hyperparameters and evaluate models.

Key Insights and Ideas

  • Ensembling for Robustness: The ensemble approach, specifically a majority vote of the top models, proved effective in improving performance over individual models, highlighting the benefit of combining diverse model strengths.
  • XLM-ROBERTa's Strength: XLM-ROBERTa (both base and large) consistently outperformed other models, suggesting its strong suitability for multilingual NER tasks, even in low-resource settings.
  • CRF Layer's Benefit: Adding a CRF layer to the transformer models generally improved performance, indicating its value in capturing sequential dependencies for NER.
  • Context Length Matters: Performance degrades significantly on very short sentences (5 words or fewer), underscoring the challenge of low-context NER. However, the model performed best on sentences with 5-10 words, showing a good balance of context.
  • Entity Distribution Impact: The models performed best on "Person" and "Location" entities, which is attributed to their higher frequency in the training data.

Key Results

The proposed ensemble model achieved a macro-averaged F1-score of 65.2% on the unofficial test dataset, securing a 20th place ranking in the multilingual track of SemEval-2022 Task 11. The ensemble model improved the F1-score of the top individual transformer-based models by approximately 1%.

Limitations and Future Work

The authors acknowledge that their results were significantly outperformed by the top-ranking systems, indicating that their approach achieved a baseline performance. They suggest that future work could explore other types of ensemble strategies, as these have shown effectiveness in improving NER model performance. The paper also implicitly highlights the challenge of very short sentences, where performance drops, suggesting further research into handling extremely limited context.

Summary updated

BibTeX
@inproceedings{rouhizadeh2022ds,
  author = {Hossein Rouhizadeh and Douglas Teodoro},
  title = {{DS4DH at SemEval-2022 Task 11: Multilingual Named Entity Recognition Using an Ensemble of Transformer-based Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure