Qtrade AI at SemEval-2022 Task 11: An Unified Framework for Multilingual NER Task

Weichao Gan, Yuanping Lin, Guangbo Yu, Guimin Chen, Qian YeNAACL 2022

TL;DR

This paper presents a unified framework for multilingual Named Entity Recognition (NER) that achieved top rankings in SemEval-2022 Task 11. The system excels by combining a robust backbone encoder with track-specific fine-tuning strategies, data augmentation for low-resource scenarios, and a novel graph-based model for Chinese NER.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and categorizing named entities (like people, organizations, locations) within text. However, real-world text is increasingly diverse, featuring multiple languages within a single document (code-mixed text) and a wide array of languages with varying linguistic characteristics. Existing NER systems often struggle with this complexity, particularly in low-resource languages or when dealing with the nuances of code-mixed data. SemEval-2022 Task 11, MultiCoNER, specifically aimed to address these challenges by focusing on semantically ambiguous, complex named entities in short, low-context, and code-mixed scenarios across 13 different tracks. This paper tackles the problem of building a single, adaptable system that can effectively perform NER across this diverse linguistic landscape.

Methods

The core of the Qtrade AI system is a unified framework built upon the XLM-ROBERTa-large backbone encoder. This choice leverages a pre-trained multilingual model to facilitate language transfer. The system employs a multi-stage fine-tuning process:

  • Full Data Fine-tuning Stage: The model is initially fine-tuned on all available data from all languages to learn general linguistic patterns.
  • Track-Specific Fine-tuning Stage: This is where the system adapts to the unique challenges of each track.
  • Chinese (ZH) Track: A novel approach, LEROBERTa-GCN4NER, is introduced. This combines a lexicon-enhanced BERT variant (LEROBERTa) with a Graph Convolutional Network (GCN) to capture lexical semantics, boundaries, and graph structures, addressing the lack of explicit word boundaries in Chinese.
  • Multilingual Track: Techniques like R-Drop (for regularization) and Shared/Private Feature Extractors are used to handle the commonalities and differences across languages.
  • Code-Mixed Track: Data augmentation techniques, specifically Mention Replacement (MR) and Shuffle within Segments (SiS), are crucial for boosting performance on this low-resource, challenging track.
  • Model Ensemble Stage: Multiple models trained with different hyperparameters are ensembled using voting to improve robustness.
  • Pseudo Label Fine-tuning Stage: Semi-supervised learning is employed by generating pseudo-labels for unlabeled data to further enhance performance.

Additionally, the system incorporates adversarial training (FGM, PGD, FreeLB) to improve robustness against perturbations.

Datasets

The paper utilizes datasets provided by the official SemEval-2022 Task 11: MultiCoNER. These datasets cover 11 languages and include tracks for:

  • Multilingual NER: Data from multiple languages where words in a sentence belong to a single language.
  • Code-Mixed NER: Data where words or phrases within a sentence can come from different languages.
  • Chinese NER: A specific track for Chinese language NER.

The datasets are described as containing around 15,000 training samples and 800 development samples per language, with six entity categories: location (LOC), person (PER), product (PROD), group (GRP), corporation (CORP), and creative works (CW).

Key Insights and Ideas

  • Unified Framework for Multilingual NER: The paper demonstrates that a single, well-designed framework can effectively handle diverse multilingual and code-mixed NER tasks, reducing the need for highly specialized models for each scenario.
  • Graph-Based Modeling for Chinese NER: The introduction of LEROBERTa-GCN4NER highlights the importance of incorporating structural linguistic information (like character relationships and lexical graphs) for languages like Chinese, which lack explicit word boundaries. This approach significantly improved performance on the Chinese track.
  • Data Augmentation is Crucial for Low-Resource Code-Mixed Data: The success on the code-mixed track underscores the effectiveness of targeted data augmentation techniques (MR and SiS) when dealing with limited labeled data. These methods help the model generalize better to unseen constructions.
  • Adversarial Training Enhances Robustness: The use of adversarial training methods improved the model's ability to handle variations and noise, contributing to better overall performance.
  • Ensembling and Pseudo-Labeling Boost Performance: Combining multiple models through ensembling and leveraging unlabeled data with pseudo-labeling are effective strategies for squeezing out the last bits of performance, especially in competitive scenarios.

Key Results

The Qtrade AI system achieved impressive results in SemEval-2022 Task 11:

  • Third place in the Multilingual Track (Subtask 11) with a macro-f1 score of 77.66.
  • Fourth place in the Code-Mixed Track (Subtask 12) with a macro-f1 score of 84.35.
  • Seventh place in the Chinese Track (Subtask 9) with a macro-f1 score of 74.00.

Specific improvements were noted:

  • The LEROBERTa-GCN4NER model achieved a 1% improvement on the Chinese dev dataset compared to its components.
  • Data augmentation techniques improved the Mix-Code track by 1.2%.
  • The combined mulsent-uni and PGD methods yielded scores of 88.46 for ZH-Chinese, 86.93 for Multilingual, and 78.21 for Mix-Code at the full data fine-tuning stage.
  • The final system achieved 5.22 points increase on the Chinese(ZH) track, 4.04 points on the Multilingual track, and 3.59 points on the Code-mixed track after pseudo-labeling.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the system performs well, it still faces challenges with semantically ambiguous entities and low-context scenarios. The effectiveness of certain techniques, like CRF, was found to be less beneficial than softmax for this specific task, possibly because pre-trained models already capture token relations well.
  • Future Work: The authors plan to further improve the system's capacity to handle polysemous scenarios and explore better methods for understanding Chinese lexicon nuances. They also aim to enhance the system's adaptability to better handle complex, ambiguous entities in challenging contexts.

Summary updated

BibTeX
@inproceedings{gan2022qtrade,
  author = {Weichao Gan and Yuanping Lin and Guangbo Yu and Guimin Chen and Qian Ye},
  title = {{Qtrade AI at SemEval-2022 Task 11: An Unified Framework for Multilingual NER Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure