MLlab4CS at SemEval-2023 Task 2: Named Entity Recognition in Low-resource Language Bangla Using Multilingual Language Models

Shrimon Mukherjee, Madhusudan Ghosh, Girish, Partha BasuchowdhuriACL 2023

TL;DR

This paper tackles the challenge of Named Entity Recognition (NER) for Bangla, a low-resource language, by leveraging multilingual transformer models. Their proposed system, MURILCRF Bangla, achieved a competitive macro F-score of 76.27% in the SemEval-2023 MultiCoNER II task, demonstrating the effectiveness of fine-tuning pre-trained models for complex NER in under-resourced linguistic contexts.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a crucial task in Natural Language Processing (NLP) that involves identifying and classifying named entities (like people, organizations, locations) within text. While NER has seen significant advancements for high-resource languages, it remains a considerable challenge for low-resource languages like Bangla. These languages often lack large, annotated datasets, making it difficult to train robust NER models. Furthermore, the SemEval-2023 MultiCoNER II task specifically focused on extracting fine-grained complex entities, adding another layer of difficulty. This paper aims to bridge this gap by developing an effective NER system for Bangla, capable of handling these complex entities, using state-of-the-art multilingual language models.

Methods

The core of the proposed system, MURILCRF Bangla, is built upon a multilingual transformer-based language model called MuRIL Large. MuRIL is specifically designed to handle the diverse linguistic characteristics of Indic languages, making it a suitable choice for Bangla.

The methodology involves a two-stage process:

  1. Pre-training and Fine-tuning: The MuRIL Large model is first pre-trained on a massive amount of unlabeled data, allowing it to learn rich linguistic, semantic, and factual information. This pre-trained model is then fine-tuned on the specific Bangla NER dataset provided for the SemEval task. This fine-tuning process adapts the general language understanding of MuRIL to the nuances of Bangla and the specific entity types required by the task.
  2. Conditional Random Field (CRF) Decoder: To further enhance the sequence labeling capabilities, a linear-chain Conditional Random Field (CRF) layer is added on top of the MuRIL model. The CRF layer helps to model the dependencies between adjacent labels, improving the overall accuracy of the entity span predictions. The system uses a negative log-likelihood loss function and Viterbi decoding for inference.

The authors also experimented with other baseline models, including IndicBertBangla and XLMR-LargeBangla, both with and without a CRF decoder, to evaluate the effectiveness of their chosen architecture.

Datasets

The primary dataset used in this work is the Bangla dataset from SemEval-2023 Task 2: MultiCoNER II. This dataset is part of a larger multilingual effort to extract fine-grained complex named entities.

  • Format: The dataset is provided in the standard CoNLL format, utilizing the BIO (Beginning-Inside-Outside) tagging scheme.
  • Structure: It consists of training, development, and testing sets. Initially, the test data only contained tokens, with tags released later for analysis.
  • Statistics: The training data contains 9,708 samples, validation data has 507 samples, and the test data has 19,859 samples.
  • Entity Types: The dataset includes 33 different fine-grained entity types, covering a wide range of categories from general ones like "ORG" and "PER" to more specific ones like "MusicalWork" and "MedicalProcedure."

Key Insights and Ideas

  • Multilingual Models for Low-Resource Languages: The success of MURILCRF Bangla highlights the power of leveraging large, pre-trained multilingual models for low-resource languages. These models, trained on vast amounts of data from multiple languages, can transfer their learned representations to languages with limited data.
  • CRF as a Complementary Layer: The addition of a CRF layer on top of a transformer model proved beneficial for improving NER performance. This suggests that while transformers excel at capturing contextual information, CRFs can effectively model the sequential dependencies crucial for accurate entity tagging.
  • Fine-Grained NER Complexity: The task's focus on fine-grained entities, as opposed to broad categories, necessitates models that can distinguish subtle differences between entity types. The chosen approach successfully addressed this complexity.
  • Performance on Low-Frequency Entities: While the model performed well overall, the entity-wise analysis revealed that it struggled with some low-frequency entities (e.g., ArtWork, AerospaceManufacture) while showing good performance on others (e.g., PrivateCORP). This points to potential areas for further improvement.

Key Results

The MURILCRF Bangla system achieved a macro average F-score of 76.27% on the test dataset for the Bangla language track of SemEval-2023 MultiCoNER II. This performance was superior to the other baseline models tested, including IndicBertBangla and XLMR-LargeBangla, both with and without CRF decoders. For instance, on the test data, MURILCRF Bangla achieved a macro F-score of 76.27%, while XLMR-LargeCRF Bangla achieved 71.74% and MURIL Bangla (without CRF) achieved 74.31%.

Limitations and Future Work

The authors acknowledge that while their model performs well, it still struggles with certain low-frequency entities.

Future work directions suggested include:

  • Few-Shot Learning: Exploring few-shot learning strategies to extract complex NERs from even more data-scarce scenarios.
  • Data-Driven Factored Modeling: Investigating data-driven factored modeling techniques to address class imbalance issues, which can arise when certain entity types are much more frequent than others.

Summary updated

BibTeX
@inproceedings{mukherjee2023mllab,
  author = {Shrimon Mukherjee and Madhusudan Ghosh and Girish and Partha Basuchowdhuri},
  title = {{MLlab4CS at SemEval-2023 Task 2: Named Entity Recognition in Low-resource Language Bangla Using Multilingual Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure