Infrrd.ai at SemEval-2022 Task 11: A system for named entity recognition using data augmentation, transformer-based sequence labeling model, and EnsembleCRF

Jianglong He, Akshay Uppal, Mamatha N, Shiv Vignesh, Deepak Kumar, Aditya Kumar SardaNAACL 2022

TL;DR

This paper tackles the challenge of Named Entity Recognition (NER) in low-resource languages, where limited training data hinders model performance. The authors propose a novel approach combining data augmentation, a multilingual transformer model fine-tuned for specific languages, and an ensemble of models with a Conditional Random Field (CRF) for optimal prediction combination, achieving competitive results in the MultiCoNER competition.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and classifying named entities (like persons, organizations, and locations) in text. While NER systems perform well in high-resource languages with abundant labeled data, they struggle significantly in low-resource languages. This scarcity of data makes it difficult for models to learn the nuances of entity recognition, especially for novel or emerging entities that are often coined or are combinations of existing words. The MultiCoNER competition specifically addresses this challenge by focusing on low-resource languages, making it a difficult benchmark for developing robust NER systems.

Methods

The core of the proposed system lies in a multi-pronged approach:

  • Data Augmentation: To combat the lack of labeled data, the authors employ several data augmentation techniques, including label-wise token replacement (LwTR), synonym replacement (SR), mention replacement (MR), and shuffle within segments (SiS). They also explore BERT-based token augmentation. The goal is to artificially increase the size and diversity of the training dataset.
  • Multilingual Transformer as a Base: Instead of training language-specific models from scratch, they leverage a pre-trained multilingual transformer model (specifically, xlm-roberta-large). This model is then fine-tuned on the available data for each target language. This transfer learning approach allows the model to benefit from knowledge learned across multiple languages.
  • EnsembleCRF: A key innovation is the EnsembleCRF architecture. This approach combines the predictions from multiple individual sequence labeling models. Instead of a simple majority vote, which can violate NER constraints (like the IOB scheme), EnsembleCRF uses a one-hot encoder to combine model outputs, followed by linear layers and a CRF layer. This allows the system to learn how to optimally combine predictions and correct errors made by individual models.

Datasets

The primary dataset used is from the MultiCoNER competition. This dataset is characterized by its low-resource nature, with a limited number of training and development sentences for each language. The authors also mention using external open-source datasets for some languages, though these did not consistently improve performance.

Key Insights and Ideas

  • Multilingual Models are Powerful Initializers: Fine-tuning a multilingual transformer model (xlm-roberta-large) proved more effective than training language-specific models from scratch or using language-specific embeddings as the primary initializer for low-resource languages.
  • Data Augmentation is Crucial, but Not a Silver Bullet: While data augmentation techniques generally improved performance, especially on the test set, their impact varied across languages. The authors found that applying a combination of augmentation techniques often outperformed single methods.
  • Ensembling is Key for Robustness: The EnsembleCRF approach significantly improved performance by learning to optimally combine predictions from multiple models, effectively mitigating individual model weaknesses and respecting NER constraints.
  • IOB Constraint Handling: The EnsembleCRF directly addresses the challenge of maintaining IOB (Inside-Outside-Beginning) tagging scheme consistency, which is a common issue with simple ensemble methods.

Key Results

The paper reports competitive results in the MultiCoNER competition, with their system ranking within the top 10 positions for most tracks. While specific macro-averaged F1 scores are presented in tables for each language, the authors highlight that their proposed method showed less than a 4% difference from the third position in many tracks. For example, in English, their best model achieved a macro-averaged F1 score of 0.7471 (Table 1, row 4).

Limitations and Future Work

  • Time and Resource Constraints: The authors acknowledge that time limitations prevented them from exploring all potential experiments, particularly for languages like Chinese. They also mention that the lack of multi-GPU support hindered their ability to conduct extensive experiments for the multilingual task.
  • Gazetteers Didn't Help: The use of gazetteers (lists of known entities) sourced from training data and Wikipedia did not yield performance improvements.
  • Future Work: The authors express interest in exploring the multilingual T5 transformer model and investigating different augmentation techniques with external data, which were not fully explored due to time constraints. They also suggest further research into understanding why data augmentation's effectiveness varies across languages.

Summary updated

BibTeX
@inproceedings{he2022infrrd,
  author = {Jianglong He and Akshay Uppal and Mamatha N and Shiv Vignesh and Deepak Kumar and Aditya Kumar Sarda},
  title = {{Infrrd.ai at SemEval-2022 Task 11: A system for named entity recognition using data augmentation, transformer-based sequence labeling model, and EnsembleCRF}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure