CMB AI Lab at SemEval-2022 Task 11: A Two-Stage Approach for Complex Named Entity Recognition via Span Boundary Detection and Span Classification

Keyu Pu, Hongyi Liu, Yixiao Yang, Jiangzhou Ji, Wenyi Lv, Yaohan HeNAACL 2022

TL;DR

This paper tackles the challenging task of multilingual complex Named Entity Recognition (NER) in short, low-context settings, particularly for code-mixed languages. They propose a novel two-stage approach: first, identifying potential entity spans, and second, classifying these spans into semantic categories. This method achieved a strong F1 score of 84.62 on a challenging sub-task, ranking third in the SemEval-2022 competition.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that aims to identify and categorize named entities (like people, organizations, locations) within text. While traditional NER has seen significant advancements, it often struggles with "complex" named entities that are semantically ambiguous or appear in contexts with limited surrounding information. The SemEval-2022 Task 11 specifically highlights this challenge by focusing on multilingual complex NER, including a particularly difficult code-mixed sub-task. Code-mixed text, where different languages are interwoven within a single sentence, further complicates NER due to linguistic variations and potential ambiguities. The paper addresses the need for robust NER systems that can handle these complexities, especially in real-world scenarios like e-commerce search queries, which are often short and code-mixed.

Methods

The authors propose a two-stage approach for complex NER:

  1. Span Boundary Detection: This first stage focuses on identifying potential spans of text that could be named entities. They achieve this by using a biaffine layer, a type of neural network layer often used in dependency parsing, connected to the last hidden states of a pre-trained language model (like XLM-ROBERTa or mT5). This model essentially learns to predict pairs of start and end tokens that likely form an entity span, treating it as a single-category NER problem.
  2. Span Classification: Once potential spans are identified, the second stage classifies these spans into their correct semantic categories (e.g., Person, Location, Organization). This is done using a classification model that takes the extracted span and its context, feeds it through a pooling layer of the pre-trained model, and then uses a fully connected layer with a softmax activation to predict the entity type.

They leverage powerful pre-trained multilingual language models, specifically XLM-ROBERTa-LARGE and mT5-LARGE, as their text encoders. To further enhance performance, they incorporate several techniques:

  • Data Augmentation: They build a dictionary of entities and randomly replace entities in the training data, then translate them to other languages to expand the corpus and improve robustness.
  • Adversarial Training: They use FGM adversarial learning to make the model more robust to slight perturbations in the input, which can help prevent misclassifications.
  • Ensemble Strategy: They combine predictions from models trained with different encoders and augmented datasets using a voting strategy to achieve the final result.

Datasets

The primary dataset used is the official raw data from SemEval-2022 Task 11. This task includes 11 base sub-tasks covering various languages (English, Spanish, Dutch, Russian, Turkish, Korean, Farsi, German, Chinese, Hindi, and Bangla) and two additional testing sets for multilingual and code-mixed queries. The paper specifically focuses on the code-mixed sub-task.

Key Insights and Ideas

  • Two-Stage is Better for Complexity: The paper strongly suggests that breaking down the complex NER problem into two distinct stages (boundary detection and classification) is more effective than a single end-to-end model, especially for ambiguous and low-context entities.
  • Biaffine Layers for Span Detection: Adapting biaffine layers, typically used for dependency parsing, for span boundary detection proves to be a successful strategy for identifying potential entity candidates.
  • Multilingual Models are Crucial: Leveraging large-scale multilingual pre-trained models like XLM-ROBERTa and mT5 is essential for handling the diverse linguistic landscape of the task.
  • Data Augmentation and Adversarial Training Pay Off: Techniques like dictionary-based data augmentation and adversarial training significantly improve model robustness and performance, particularly in data-scarce or challenging scenarios.
  • Ensembling for the Win: Combining the strengths of multiple models through an ensemble strategy is a key factor in achieving top performance, highlighting the benefit of diverse model predictions.

Key Results

The proposed two-stage approach, combined with various optimization techniques and an ensemble strategy, achieved a significant F1 score of 84.62 on the code-mixed sub-task of SemEval-2022 Task 11. This result ranked the CMB AI Lab team third in the competition. The paper also shows incremental improvements from baseline models to their final ensemble, demonstrating the effectiveness of each added technique. For instance, starting from a baseline XLM-ROBERTa+CRF F1 of 79.7, their two-stage approach with various enhancements reached 84.62.

Limitations and Future Work

  • Limitations: The paper notes that while their approach is effective, the performance on the code-mixed sub-task is still lower than on some of the monolingual sub-tasks, indicating that code-mixing remains a significant challenge. They also mention that for enhanced data, a weight less than 1 is typically given, implying that the exact weighting strategy for augmented data might be a point of fine-tuning.
  • Future Work: The authors suggest that further exploration of ensemble strategies and potentially more advanced data augmentation techniques could lead to even better results. The robustness gained through adversarial training also opens doors for applying this method to other challenging NLP tasks with noisy or limited data.

Summary updated

BibTeX
@inproceedings{pu2022cmb,
  author = {Keyu Pu and Hongyi Liu and Yixiao Yang and Jiangzhou Ji and Wenyi Lv and Yaohan He},
  title = {{CMB AI Lab at SemEval-2022 Task 11: A Two-Stage Approach for Complex Named Entity Recognition via Span Boundary Detection and Span Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure