silpa_nlp at SemEval-2022 Tasks 11: Transformer based NER models for Hindi and Bangla languages

Sumit Singh, Pawankumar Jawale, Uma TiwaryNAACL 2022

TL;DR

This paper explores the effectiveness of various Transformer-based models for Named Entity Recognition (NER) on Hindi and Bangla languages. The authors found that the MuRIL (Large) model significantly outperformed others, highlighting the importance of models specifically pre-trained on Indic languages for achieving state-of-the-art results in this domain.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and classifying named entities (like persons, locations, organizations) within text. While Transformer models have revolutionized NLP, their performance on low-resource languages, particularly for complex NER tasks, remains an active area of research. This paper specifically addresses the challenge of applying and evaluating Transformer-based NER models for Hindi and Bangla within the context of the SemEval-2022 Task 11 (MultiCoNER), which focuses on multilingual complex named entity recognition. The goal is to determine which pre-trained Transformer models are best suited for this task and to analyze their performance on these Indic languages.

Methods

The researchers employed a fine-tuning approach for several Transformer-based pre-trained models. The core methodology involved taking existing models and adapting them to the specific NER task for Hindi and Bangla. The models explored include:

  • mBERT: A multilingual BERT model pre-trained on 104 languages.
  • IndicBERT: A model pre-trained on 11 Indian languages.
  • XLM-ROBERTa (Base and Large): Multilingual models pre-trained on 100 languages.
  • MuRIL (Base and Large): Models specifically pre-trained on 17 Indic languages, augmented with monolingual corpora and translated/transliterated document pairs.

The fine-tuning process involved adding a fully connected layer on top of the Transformer's output to map token embeddings to the NER labels. Standard techniques like AdamW optimizer, cross-entropy loss, and dynamic learning rate scheduling were used. Preprocessing involved subword tokenization (SentencePiece for XLM-ROBERTa and IndicBERT, WordPiece for mBERT and MuRIL) and label alignment for subword tokens. Post-processing involved realigning predicted labels to the original sentence structure.

Datasets

The study utilized datasets provided for SemEval-2022 Task 11, specifically for the Hindi and Bangla tracks. These datasets were divided into training, development, and testing sets. The NER entities covered include Location (LOC), Person (PER), Production (PROD), Group (GRP), Corporation (CORP), and Creative Work (CW). The data uses the BIO (Beginning-Inside-Outside) tagging scheme. The paper notes that for both Hindi and Bangla, the training and development datasets contained tokens with tags, while the testing datasets only provided tokens.

Key Insights and Ideas

  • Indic-specific pre-training matters: The most significant finding is that models pre-trained specifically on Indic languages (MuRIL) significantly outperform general multilingual models (mBERT, XLM-ROBERTa) for NER in Hindi and Bangla. This underscores the importance of linguistic and cultural context embedded in pre-training data.
  • Larger models generally perform better: Across most model families (e.g., XLM-ROBERTa Base vs. Large, MuRIL Base vs. Large), the larger variants demonstrated superior performance, suggesting that increased model capacity is beneficial for capturing complex linguistic nuances.
  • Creative Work (CW) is the most challenging entity: The analysis revealed that predicting Creative Work entities was the most difficult for all models, indicating a potential area for future research and model improvement. Conversely, Person (PER) entities were generally easier to predict.

Key Results

The study evaluated six Transformer-based models on the Hindi and Bangla NER tasks. The MuRIL (Large) model achieved the highest overall F1-scores, with 0.69 for Hindi and 0.59 for Bangla. MuRIL (Base) was the second-best performing model. The paper also provides detailed entity-wise F1-scores, showing MuRIL (Large) leading across most entity types for both languages. A qualitative analysis of a sample sentence demonstrated that MuRIL's output was closer to the gold annotation compared to other models.

Limitations and Future Work

The authors acknowledge that predicting Creative Work (CW) entities is particularly challenging. While they don't explicitly detail future work, the findings strongly suggest that developing more specialized pre-trained models for Indic languages, or further enhancing existing ones like MuRIL, would be a promising direction. Exploring more advanced techniques for handling complex entity types like CW could also be a valuable avenue for future research.

Summary updated

BibTeX
@inproceedings{singh2022silpa,
  author = {Sumit Singh and Pawankumar Jawale and Uma Tiwary},
  title = {{silpa\_nlp at SemEval-2022 Tasks 11: Transformer based NER models for Hindi and Bangla languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure