Bangor University at WojoodNER 2024: Advancing Arabic Named Entity Recognition with CAMeLBERT-Mix

Norah AlshammariACL 2024

TL;DR

This paper presents Bangor University's approach to the WojoodNER 2024 shared task, focusing on fine-grained Arabic Named Entity Recognition (NER). They achieved competitive results by fine-tuning the CAMeLBERT-Mix transformer model with a custom linear classification layer, demonstrating the effectiveness of advanced pre-trained models for complex Arabic NLP tasks.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and categorizing entities like people, organizations, and locations within text. Arabic, with its rich morphology, dialectal variations, and lack of capitalization, presents unique challenges for NER. The WojoodNER 2024 shared task specifically targets "fine-grained" NER, meaning it requires identifying a large number of specific entity types and subtypes, making it a particularly difficult problem. Existing Arabic NER datasets have often focused on simpler entity structures or a limited set of entity types. This paper addresses the need for robust Arabic NER systems capable of handling this increased granularity and complexity.

Methods

The core of Bangor University's system is the CAMeLBERT-Mix model, a transformer-based architecture specifically pre-trained on a diverse Arabic corpus. This model was then fine-tuned on the Wojood-Fine dataset. A key innovation is the addition of a linear classification layer on top of the pre-trained CAMeLBERT-Mix. This layer takes the contextualized embeddings generated by the transformer and maps them to the 51 different entity types and subtypes (plus a non-entity 'O' label) required by the task.

The preprocessing pipeline involved using the tokenizer accompanying CAMeLBERT-Mix, which is designed to handle Arabic's subword complexities. They also incorporated standard NLP practices like adding [CLS] and [SEP] tokens for sentence boundaries.

For training, they used a batch size of 32, 50 epochs, and the AdamW optimizer with a learning rate of 1e-4. A StepLR scheduler was employed to adjust the learning rate, and Cross-Entropy Loss was used for multi-class classification. Crucially, they implemented early stopping based on validation loss to prevent overfitting.

Datasets

The primary dataset used is the Wojood-Fine corpus, introduced by Liqreina et al. (2023). This corpus is specifically designed for fine-grained Arabic NER.

  • Training Data: 70% of the Wojood-Fine corpus.
  • Validation Data: 10% of the Wojood-Fine corpus, used for tuning model parameters and monitoring performance during training.
  • Testing Data: 20% of the Wojood-Fine corpus, used for final, blind evaluation to ensure impartial assessment.

The paper also references previous Arabic NER datasets like ANERCorp, ACE2005, and the CANER Corpus, highlighting the evolution towards more comprehensive and fine-grained annotations.

Key Insights and Ideas

  • Transformer Power for Arabic: The success of CAMeLBERT-Mix demonstrates that advanced, pre-trained transformer models are highly effective for capturing the nuanced linguistic structures of Arabic, even for complex, fine-grained tasks.
  • Linear Layer Simplicity and Effectiveness: Adding a simple linear layer on top of a powerful pre-trained model proved sufficient to map rich contextual embeddings to a large number of specific entity classes. This suggests that much of the heavy lifting is done by the pre-trained model's feature extraction capabilities.
  • Fine-Grained NER is Challenging: The drop in performance from training to validation metrics highlights the inherent difficulty of fine-grained NER, where distinguishing between numerous, sometimes overlapping, entity types requires robust generalization.
  • Importance of Validation for Overfitting: The explicit mention of monitoring validation loss and using early stopping underscores the critical need to prevent models from memorizing the training data, especially in tasks with many classes and potentially imbalanced data.

Key Results

The Bangor University system achieved competitive results in the WojoodNER 2024 competition. For the Flat Fine-Grain NER subtask (Subtask-1), their system achieved:

  • Precision: 88.06%
  • Recall: 84.90%
  • F1-Score: 86.45%

These scores placed them 5th among the participating teams in Subtask-1. The paper also shows results for a nested NER subtask, where their baseline achieved an F1-score of 92%. The results indicate a strong performance, with precision and recall values being relatively balanced.

Limitations and Future Work

  • Overfitting Concerns: While the model performed well, the noticeable drop in metrics from training (F1: 95.87%) to validation (F1: 89.16%) suggests potential for overfitting.
  • Future Work Suggestions: The authors suggest exploring techniques like dropout, regularization, and data augmentation to further bridge the gap between training and validation performance and improve generalization. They also plan to refine their training strategies and preprocessing methods for consistent performance improvements.

Summary updated

BibTeX
@inproceedings{alshammari2024bangor,
  author = {Norah Alshammari},
  title = {{Bangor University at WojoodNER 2024: Advancing Arabic Named Entity Recognition with CAMeLBERT-Mix}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure