DANGNT-SGU at SemEval-2022 Task 11: Using Pre-trained Language Model for Complex Named Entity Recognition

Dang Nguyen, Huy Khac Nguyen HuynhNAACL 2022

TL;DR

This paper presents a system for recognizing complex and ambiguous named entities, a challenging task in NLP. The authors leverage pre-trained language models (PLMs) like BERT and RoBERTa, fine-tuning them on the SemEval-2022 Task 11 dataset. Their findings highlight the effectiveness of PLMs, particularly XML-RoBERTa, in this domain, while also noting the inherent difficulties of the task.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that aims to identify and categorize entities like people, organizations, and locations within text. However, traditional NER models struggle with "complex named entities." These are entities that are not simple proper nouns, such as titles of creative works (e.g., "Eternal Sunshine of the Spotless Mind"), gerunds used as titles (e.g., "Saving Private Ryan"), or even full clauses functioning as names. The SemEval-2022 Task 11 specifically targets this challenge, focusing on semantically complex and ambiguous entities that are often difficult for existing models to recognize accurately. This difficulty can significantly impact the performance of downstream NLP applications.

Methods

The core of the authors' approach relies on Pre-trained Language Models (PLMs). They specifically utilize models based on the BERT architecture, which has demonstrated state-of-the-art performance on various NLP tasks. The process involves fine-tuning these pre-trained models on the specific dataset for the complex NER task.

The authors experimented with several PLMs available through the HuggingFace Transformers library:

  • BERT (base and large, uncased versions): A foundational transformer model pre-trained on a large English corpus.
  • RoBERTa (large version): An optimized version of BERT that improves training strategies, such as dynamic masking and larger batch sizes.
  • XLM-RoBERTa (large version): A multilingual variant of RoBERTa pre-trained on a massive dataset spanning 100 languages, making it suitable for cross-lingual tasks.

They also explored adding a linear and softmax layer on top of the PLMs for entity recognition, but found it did not significantly improve results. The data was formatted according to the CONLL format, a standard for NER tasks.

Datasets

The primary dataset used in this paper is the SemEval-2022 Task 11: MultiCoNER Multilingual Complex Named Entity Recognition dataset. The authors focused on the English track of this dataset. They utilized:

  • Training data: 15,300 examples.
  • Development/Practice Phase data: 800 examples.
  • Evaluation Phase data: A significantly larger dataset of 217,818 examples, presenting a substantial challenge.

The dataset includes six entity types: Person (PER), Location (LOC), Group (GRP), Corporation (CORP), Product (PROD), and Creative Work (CW).

Key Insights and Ideas

  • PLMs are powerful for complex NER: The study strongly supports the idea that leveraging large pre-trained language models is crucial for tackling the complexities of recognizing non-standard named entities.
  • XLM-RoBERTa shines: Among the tested models, XLM-RoBERTa (large version) consistently delivered strong performance, particularly excelling in recognizing Creative Works, Groups, and Persons.
  • Subtle differences matter: While the performance differences between the top models were not drastic, they were significant enough to influence model selection for the final evaluation.
  • Hyperparameter tuning has limits: The authors found that extensive hyperparameter tuning did not yield substantial improvements, suggesting that the inherent complexity of the task might be a limiting factor.

Key Results

The paper reports evaluation results in terms of Macro Precision, Recall, and F1-score for each entity type.

In the Practice Phase:

  • XML-RoBERTa-large and BERT-large-uncased emerged as the top performers, outperforming the other models across most entity types. XML-RoBERTa-large was particularly strong in PER, CW, and GRP.
  • BERT-base-uncased surprisingly performed best for the PROD entity type.

In the Evaluation Phase, where the dataset was much larger:

  • The authors selected the XML-RoBERTa-large model for the final submission.
  • The overall F1-score for the Evaluation Phase was 0.67.
  • Specific F1-scores for the Evaluation Phase included: PER (0.8474), LOC (0.6866), CW (0.5641), GRP (0.6448), CORP (0.6448), and PROD (0.6258).

Limitations and Future Work

Limitations: * The authors acknowledge that despite using advanced PLMs, the results in the Evaluation Phase were not as good as anticipated, indicating the significant challenge posed by the larger dataset. * The attempt to improve results by adding a linear and softmax layer on top of the PLMs did not yield significant improvements.

Future Work: * The authors plan to continue testing other PLMs. * They intend to explore other deep learning techniques and theories to further improve accuracy in recognizing complex entities.

Summary updated

BibTeX
@inproceedings{nguyen2022dangnt,
  author = {Dang Nguyen and Huy Khac Nguyen Huynh},
  title = {{DANGNT-SGU at SemEval-2022 Task 11: Using Pre-trained Language Model for Complex Named Entity Recognition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure