CSECU-DSG at SemEval-2022 Task 11: Identifying the Multilingual Complex Named Entity in Text Using Stacked Embeddings and Transformer based Approach

Abdul Aziz, Md. Akram Hossain, Abu Nowshed ChyNAACL 2022

TL;DR

This paper tackles the challenging task of identifying complex and ambiguous named entities across multiple languages. The authors propose two approaches: a BiLSTM-CRF model with stacked embeddings and a transformer-based approach using pre-trained models like BERT and XLM-RoBERTa. Their work contributes to the SemEval-2022 MultiCoNER shared task, demonstrating competitive performance, particularly with transformer models, on this multilingual NER problem.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), crucial for applications like search and data mining. Traditionally, NER focused on identifying straightforward entities like person names, locations, and organizations. However, the explosion of unstructured social media data has introduced a new challenge: identifying complex and ambiguous named entities. These entities are often embedded within noisy text, exhibit linguistic variations, and can be multilingual. The SemEval-2022 Task 11, "MultiCoNER," was specifically designed to address this by focusing on complex named entity recognition in multilingual settings, including monolingual, multilingual, and code-mixed tracks.

Methods

The CSECU-DSG team explored two primary approaches:

  • BiLSTM-CRF with Stacked Embeddings: This method leverages the well-established BiLSTM-CRF architecture for sequence labeling. The novelty lies in their use of a "stacked embedding strategy," inspired by the Flair framework. This involves concatenating embeddings from various pre-trained models (e.g., GloVe, FastText, character embeddings) to create richer representations for each token. These enhanced embeddings are then fed into the BiLSTM encoder, followed by a CRF layer for final predictions.
  • Transformer-based Approach: This approach utilizes powerful pre-trained transformer models. For individual languages, they employed monolingual BERT models. For multilingual and code-mixed tracks, they opted for XLM-RoBERTa, a robust cross-lingual model known for its strong performance across various languages without extensive task-specific fine-tuning. The paper highlights that XLM-RoBERTa often outperformed monolingual BERT, especially in multilingual scenarios.

Datasets

The primary dataset used for this task is the MultiCoNER dataset, introduced by the SemEval-2022 Task 11 organizers. This dataset is notable for its multilingual nature, covering 11 languages (Bangla, German, English, Spanish, Farsi, Hindi, Korean, Dutch, Russian, Turkish, Chinese) and also including a code-mixed dataset. The paper provides statistics on the training, validation, and test splits for each language track.

Key Insights and Ideas

  • XLM-RoBERTa's Cross-Lingual Prowess: The paper strongly suggests that XLM-RoBERTa is a highly effective choice for multilingual and code-mixed NER tasks. Its ability to leverage knowledge learned from one language to perform well in others without additional training is a significant advantage.
  • Stacked Embeddings Offer Richness: While transformer models often dominate, the exploration of stacked embeddings with BiLSTM-CRF shows that combining diverse embedding sources can still yield competitive results, particularly in capturing nuanced contextual information.
  • Complexity of Multilingual NER: The results highlight that even with advanced models, achieving high performance across all languages remains a challenge. Some languages, like Chinese and Hindi, saw better performance from the proposed systems, while others lagged behind top participants.

Key Results

The paper presents results for both their BiLSTM-CRF and transformer-based systems across various tracks.

  • Transformer-based System Dominance: In most languages, the transformer-based approach, particularly using XLM-RoBERTa, achieved better macro F1-macro scores compared to the BiLSTM-CRF system.
  • Precision vs. Recall Trade-off: While the transformer system generally led in F1-macro, the BiLSTM-CRF system sometimes showed higher precision, and the transformer system often achieved better recall. This indicates different strengths and weaknesses between the two architectures.
  • Competitive Performance in Specific Languages: The CSECU-DSG system achieved competitive results, ranking 9th for Chinese (ZH) with an F1-macro of 0.6722 and 11th for Hindi (HI) with an F1-macro of 0.5768, placing them among the top performers in these specific language tracks.

Limitations and Future Work

  • Performance Gaps: The authors acknowledge that their proposed systems did not perform optimally across all languages, indicating room for improvement.
  • Future Directions: The team plans to incorporate task-specific features and explore existing NER technologies to create a more unified architecture. They aim to better address the challenges of complex and multilingual NER by fusing these technologies.

Summary updated

BibTeX
@inproceedings{aziz2022csecu,
  author = {Abdul Aziz and Md. Akram Hossain and Abu Nowshed Chy},
  title = {{CSECU-DSG at SemEval-2022 Task 11: Identifying the Multilingual Complex Named Entity in Text Using Stacked Embeddings and Transformer based Approach}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure