LMN at SemEval-2022 Task 11: A Transformer-based System for English Named Entity Recognition

Ngoc LaiNAACL 2022

TL;DR

This paper presents a simple yet effective Transformer-based system for complex Named Entity Recognition (NER) that achieved a competitive 12th place out of 30 teams in SemEval-2022 Task 11. The authors explored a data augmentation technique using entity linking, which, while not improving final performance, offers a promising avenue for future research.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), aiming to identify and categorize named entities (like people, organizations, and locations) in text. However, recognizing "complex" named entities poses a significant challenge. These entities can be more than just simple noun phrases; they can include gerunds, infinitives, or even full clauses (e.g., "National Baseball Hall of Fame and Museum," "Saving Private Ryan," "I Capture The Castle"). The ambiguity and varied structure of these complex entities make them difficult to identify accurately, especially in low-context situations. This paper addresses the challenge of recognizing these complex named entities, particularly within the context of the SemEval-2022 Task 11, which focuses on multilingual complex NER.

Methods

The core of the proposed system is a Transformer-based encoder (specifically, a BERT-large model) fine-tuned for sequence labeling. This approach leverages the power of pre-trained language models to capture contextual information effectively. The authors formulate the task as predicting a label for each input token, using the BIO (Beginning, Inside, Outside) tagging scheme.

A key aspect of their method is the absence of a Conditional Random Field (CRF) layer, which is common in many sequence labeling models. They cite recent work suggesting that for Transformer-based models, a CRF layer might not significantly improve performance.

Additionally, the paper explores a data augmentation technique using entity linking. This involves:

  1. Identifying a named entity in a sentence.
  2. Using an "entity linker" to connect this entity to its corresponding entry in a knowledge graph (like Wikidata).
  3. Extracting information about the original entity from the knowledge graph.
  4. Finding a new entity that is similar (e.g., of the same type) but different from the original.
  5. Replacing the original entity with the new one to create a synthetic training example. This aims to expose the model to more variations of entities.

Datasets

The primary dataset used for this work is the English track of SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER). The paper mentions using this dataset for both training and evaluation.

Key Insights and Ideas

  • Simplicity can be effective: The paper demonstrates that a straightforward Transformer-based approach, without complex additions like a CRF layer, can achieve competitive results in a challenging NER task. This highlights the power of modern pre-trained language models.
  • Data augmentation's potential: While their entity linking-based data augmentation didn't improve performance in this specific experiment, the authors believe it holds significant potential. This suggests that generating more diverse and contextually relevant training data could be crucial for tackling complex NER.
  • Transformer models are powerful for sequence labeling: The success of their baseline model underscores the suitability of Transformer architectures for sequence labeling tasks, even for complex entity types.

Key Results

The proposed Transformer-based system achieved a macro F1 score of 72.50% on the held-out test set, ranking 12th out of 30 participating teams in the SemEval-2022 Task 11 English track.

When evaluating different Transformer models on the development set, ALBERT-xxlarge trained for 20 epochs showed the best performance. However, the final system used for prediction was based on this model, trained on the original training set without data augmentation.

The data augmentation approach, when applied to a RoBERTa-large model, resulted in worse performance compared to the model trained solely on the original data.

Limitations and Future Work

The authors acknowledge that the data augmentation approach, while promising, did not yield performance improvements in their experiments. They suggest that further exploration is needed to refine this technique.

Future work includes:

  • Conducting a thorough error analysis to better understand model weaknesses.
  • Applying visualization techniques to gain deeper insights into model behavior.
  • Investigating methods to reduce the computational complexity of Transformer models, as they are computationally expensive.

Summary updated

BibTeX
@inproceedings{lai2022lmn,
  author = {Ngoc Lai},
  title = {{LMN at SemEval-2022 Task 11: A Transformer-based System for English Named Entity Recognition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure