VBD_NLP at SemEval-2023 Task 2: Named Entity Recognition Systems Enhanced by BabelNet and Wikipedia

Phu Gia Hoang, Le Thanh, Hai-Long TrieuACL 2023

TL;DR

This paper tackles the challenge of fine-grained Named Entity Recognition (NER) in English and Bangla, particularly for low-resource languages, by augmenting limited training data with knowledge from BabelNet and Wikipedia. Their approach significantly boosts performance, especially for Bangla, demonstrating the power of knowledge-based data augmentation for complex NER tasks.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and classifying named entities (like people, organizations, locations) in text. The SemEval-2023 Task 2, "MultiCoNER II," presented a significant challenge: identifying a large number of fine-grained named entity types (35 in total) with only a small amount of training data. This scarcity of labeled data is a common hurdle, especially for low-resource languages like Bangla, where linguistic resources are often limited. Existing models struggle to generalize effectively when faced with both a high number of entity classes and insufficient training examples. This paper aims to bridge this gap by developing robust NER systems that can handle this complexity, particularly for English and Bangla.

Methods

The core of the VBD_NLP system lies in its data augmentation strategy, leveraging external knowledge bases.

  • Knowledge-Based Data Augmentation: The authors employed a two-pronged approach to enrich their training data:
  • Wikipedia Redirections: They used Wikipedia redirection links to identify synonymous terms and link them to their corresponding Wikipedia articles. This helps in capturing variations in how entities are mentioned.
  • BabelNet Concepts: They utilized BabelNet, a large multilingual lexicalized knowledge base, to identify high-level concepts and their hyponyms (child synsets). This process involved retrieving concepts and their instances from Wikipedia articles, then filtering and annotating them. For Bangla, they leveraged English-Bangla mappings within BabelNet.
  • Self-Annotating Loop: To further refine the augmented data and improve its quality, they implemented a self-annotating loop inspired by Tedeschi and Navigli (2022). This iterative process involves training a model on a subset of the data, using it to annotate another subset, and then filtering out noisy or incorrect annotations to create a cleaner, larger dataset for subsequent training.
  • Model Architecture: The underlying NER model is built upon the powerful mDeBERTaV3 pre-trained language model. This is combined with a BiLSTM-CRF (Bidirectional Long Short-Term Memory with Conditional Random Field) architecture, a standard and effective approach for sequence tagging tasks like NER, to enhance token representations before final label prediction.

Datasets

The primary datasets used were from the SemEval-2023 Task 2: MultiCoNER II.

  • MultiCoNER II: This dataset is an extension of the previous MultiCoNER task, featuring a more complex taxonomy with 35 fine-grained entity types across 12 languages. The task is characterized by a significant imbalance between the size of the training and testing sets, with test sets being considerably larger to evaluate generalization capabilities.
  • English and Bangla Subtracks: The authors focused on the English and Bangla subtracks of this dataset. They highlight that Bangla is a low-resource language, making data augmentation particularly crucial.

Key Insights and Ideas

  • Knowledge Augmentation is Key for Low-Resource NER: The most striking insight is the significant performance boost achieved for Bangla, a low-resource language. This underscores the effectiveness of leveraging external knowledge bases like BabelNet and Wikipedia when labeled data is scarce.
  • BabelNet and Wikipedia Synergy: The combination of BabelNet's conceptual hierarchy and Wikipedia's vast article content proved to be a powerful strategy for generating diverse and relevant training examples.
  • Self-Annotation for Data Quality: The self-annotating loop is crucial for mitigating the noise introduced during automatic annotation, ensuring that the augmented data is not only larger but also of higher quality.
  • Language-Specific Performance Differences: While augmentation helped both languages, the impact was more pronounced in Bangla. This suggests that the inherent complexity and grammatical structure of English might pose different challenges for augmentation strategies compared to Bangla.

Key Results

The augmented systems demonstrated substantial improvements over the baseline models:

  • English: The augmented system achieved a macro-f1 score of 52.64%, an improvement of 2.38% over the baseline. Precision saw a notable increase (57.89% vs. 51.64%), while recall remained similar.
  • Bangla: The performance gains were even more dramatic for Bangla. The augmented system achieved a macro-f1 score of 64.31%, an improvement of 11.33% over the baseline. Precision, recall, and F1-score all saw significant increases, with precision jumping by over 10%.
  • Fine-grained Performance: Detailed analysis across various fine-grained entity types (Tables 5, 6, 9, 10) shows that the augmented model often outperforms the baseline, especially in Bangla, across many categories. However, in English, the baseline model sometimes performed better in specific classes, suggesting that augmentation might not uniformly benefit all entity types or that the English language's complexity presents unique challenges.

Limitations and Future Work

  • Limited Language Scope: The study focused solely on English and Bangla. The authors acknowledge the need to extend their experiments to other languages, particularly other low-resource ones.
  • Noise Filtering: While the self-annotating loop helps, further improvements in filtering noisy annotations are identified as a key area for future work.
  • Complex Entity Types: The paper suggests exploring methods for extracting more complex entities, such as nested entities, which were not the primary focus of this work.
  • English vs. Bangla Nuances: The paper notes that the augmentation's impact differed between English and Bangla, hinting at the need for language-specific tuning or more sophisticated augmentation strategies that account for linguistic complexities.

Summary updated

BibTeX
@inproceedings{hoang2023vbd,
  author = {Phu Gia Hoang and Le Thanh and Hai-Long Trieu},
  title = {{VBD\_NLP at SemEval-2023 Task 2: Named Entity Recognition Systems Enhanced by BabelNet and Wikipedia}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure