Multilinguals at SemEval-2022 Task 11: Complex NER in Semantically Ambiguous Settings for Low Resource Languages

Amit Pandey, Swayatta Daw, Narendra Unnam, Vikram PudiNAACL 2022

TL;DR

This paper tackles the challenging task of Named Entity Recognition (NER) for low-resource languages (Spanish and Chinese) with semantically ambiguous entities. The authors demonstrate that a relatively simple BERT+CRF model, enhanced with Whole Word Masking during pre-training, significantly outperforms baselines and achieves competitive results on a challenging benchmark.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and classifying named entities (like people, organizations, locations) in text. While NER for high-resource languages like English has seen significant progress, it remains a challenge for low-resource languages due to the scarcity of labeled data. Furthermore, the task becomes even more complex when dealing with "complex NER," which includes entities with ambiguous textual structures (e.g., Creative Works, Products) and semantically ambiguous contexts, often found in short search queries or low-context sentences. This paper specifically addresses the need for robust NER systems in these challenging low-resource, semantically ambiguous settings, as defined by the SemEval-2022 Task 11.

Methods

The core of the proposed solution lies in leveraging pre-trained language models, specifically BERT, and adapting them for the complex NER task. The authors employ two key strategies:

  • Whole Word Masking (WWM): For Chinese, which lacks explicit word boundaries, they use Chinese Word Segmentation (CWS) to split text into words before applying WWM. This strategy, where entire words are masked instead of sub-word units, is shown to improve the performance of the masked language modeling objective on large, unsupervised corpora. For Spanish, they adopt a similar WWM strategy.
  • Model Architectures: They experiment with three architectures built upon a fine-tuned BERT layer:
  • BERT+CRF: This model uses BERT embeddings and then applies a Conditional Random Field (CRF) layer. CRFs are effective for sequence labeling tasks as they capture inter-token dependencies, mitigating label bias issues common in simpler softmax classifiers.
  • BERT+BiLSTM+CRF: This architecture adds a Bidirectional Long Short-Term Memory (BiLSTM) layer between the BERT embeddings and the CRF layer. The BiLSTM aims to extract richer contextual information.
  • BERT+Linear: This is the simplest model, using BERT embeddings followed by fully connected layers and a softmax classifier.

Datasets

The primary dataset used is the MultiCoNER dataset. This dataset is specifically designed for multilingual complex NER and includes data for several low-resource languages. The authors focus on the Spanish and Chinese portions of this dataset for their experiments. The dataset follows a BIO tagging scheme and includes six entity types: Person (PER), Location (LOC), Group (GRP), Corporation (CORP), Product (PROD), and Creative Work (CW). They utilize the training and development sets for model training and validation, and the blind test set for final evaluation.

Key Insights and Ideas

  • Simplicity Can Be Powerful: The paper highlights that a simpler BERT+CRF model can outperform more complex architectures like BERT+BiLSTM+CRF, especially in low-resource settings where overfitting is a significant concern. The BiLSTM, while capturing more context, might be too heavy for the limited training data.
  • Whole Word Masking is Crucial for Chinese: The adaptation of Whole Word Masking, combined with Chinese Word Segmentation, proves to be a vital step for improving BERT's performance on Chinese text, which is inherently different from alphabetic languages.
  • CRF's Value in Complex NER: The inclusion of a CRF layer consistently improves performance over a simple linear classifier, demonstrating its effectiveness in modeling the sequential nature of NER and capturing dependencies between entities.
  • Long-Tail Entities are the Bottleneck: The error analysis reveals that entities with lower frequency (long-tailed entities like GRP, PROD, CW) are the most challenging to predict, confirming the hypothesis that their rarity in training data makes them harder for models to learn.

Key Results

  • The BERT+CRF model emerged as the best-performing model across both Spanish and Chinese languages on the validation set.
  • On the blind test set, the BERT+CRF model achieved an F1 score of 0.6120 for Spanish and 0.6695 for Chinese.
  • The proposed approach significantly outperformed the baseline (XLM-ROBERTa) by approximately 5% for Spanish and 10% for Chinese in terms of F1 score.
  • The BERT+CRF model achieved competitive positions on the evaluation leaderboard for the shared task.

Limitations and Future Work

  • Generalization to Test Set: The authors note a drop in performance on the blind test set compared to the validation set. They attribute this to the model's potential difficulty in generalizing to the specific nature of the test set, which includes questions and short search queries.
  • Handling Long-Tailed Entities: The error analysis clearly indicates that predicting rare, long-tailed entities remains a significant challenge.
  • Future Work: The authors plan to leverage their findings to address label scarcity in other low-resource languages and code-mixed data. They also aim to explore further improvements for handling complex entity types and semantically ambiguous contexts.

Summary updated

BibTeX
@inproceedings{pandey2022multilinguals,
  author = {Amit Pandey and Swayatta Daw and Narendra Unnam and Vikram Pudi},
  title = {{Multilinguals at SemEval-2022 Task 11: Complex NER in Semantically Ambiguous Settings for Low Resource Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure