LSJSP at SemEval-2023 Task 2: FTBC: A FastText based framework with pre-trained BERT for NER

Shilpa Chatterjee, Leo Evenss, Pramit Bhattacharyya, Joydeep MondalACL 2023

TL;DR

This paper introduces FTBC, a novel framework for Named Entity Recognition (NER) that combines FastText for spell correction with pre-trained BERT models for robust entity identification, particularly on noisy, multilingual datasets. FTBC significantly outperforms baseline BERT-CRF models across 12 languages in the SemEval-2023 Task 2, demonstrating its effectiveness in handling complex entities and simulated errors.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a crucial task in Natural Language Processing (NLP) for identifying and classifying entities like people, locations, and organizations within text. While pre-trained models like BERT have greatly advanced NER performance, they often struggle with low-resource languages, noisy datasets (e.g., typos, spelling mistakes), and complex entity types. The SemEval-2023 Task 2: MultiCoNER II specifically challenges researchers to build NER systems for 12 languages, including simulated errors, making it a realistic testbed for robust NER. Existing models often fall short when faced with these combined challenges, necessitating more resilient approaches.

Methods

The proposed FTBC framework tackles the problem in two stages. First, it employs a FastText-based spell checker to pre-process the input text. This layer leverages FastText's ability to find nearest neighbors for potentially misspelled words. To mitigate the issue of FastText sometimes returning "garbage values," the authors incorporate a Levenshtein distance threshold (set at 50) to decide whether to accept the FastText suggestion. If the distance is too large, the original token is retained.

The output of this spell-checking layer is then fed into a BERT-CRF model. This second layer uses pre-trained BERT embeddings to capture contextual information and a Conditional Random Field (CRF) layer to model the dependencies between adjacent tags, improving sequence labeling accuracy. The BERT-CRF model is fine-tuned on the provided NER dataset. This two-layered approach aims to first clean the input data and then leverage powerful pre-trained models for accurate entity recognition.

Datasets

The primary dataset used is the MultiCoNER II dataset, which was created for the SemEval-2023 Task 2. This dataset covers 12 languages (Bangla, German, English, Spanish, Farsi, French, Hindi, Italian, Portuguese, Swedish, Ukrainian, and Chinese). A key characteristic of MultiCoNER II is the inclusion of simulated errors, such as typographical and spelling mistakes, to mimic real-world noisy text. The dataset features a fine-grained classification of over 30 entity classes, which are mapped from six broader categories.

Key Insights and Ideas

  • The synergy of FastText and BERT is powerful: Combining a fast, subword-aware spell checker like FastText with a powerful contextual model like BERT proves highly effective, especially for noisy data.
  • Levenshtein distance as a quality filter: Using Levenshtein distance to filter FastText's nearest neighbor suggestions is a practical way to prevent the introduction of incorrect corrections, maintaining data integrity.
  • Multilingual NER is challenging but achievable: The FTBC framework demonstrates strong performance across a diverse set of 12 languages, highlighting the potential for generalizable NER solutions.
  • Ambiguity is a persistent challenge: Even with advanced models, ambiguous entities (like "John" being classified as Politician, OtherPER, or Artist) remain a significant hurdle, suggesting that external knowledge or more sophisticated disambiguation techniques might be needed.

Key Results

The FTBC system achieved an average F1 score of 58.27% on fine-grained entities and 75.79% on coarse-grained entities across all 12 monolingual tracks of the MultiCoNER II task. Crucially, FTBC outperformed the baseline BERT-CRF model in all 12 monolingual tracks for both fine and coarse-grained categories. The paper also notes that fine-grained categories like PrivateCorp, Scientist, and Symptom were frequently misclassified, and categories like AerospaceManufacturer had very low F1 scores, suggesting areas for improvement.

Limitations and Future Work

The authors acknowledge that ambiguous entities continue to affect the overall performance of their system, even with the FTBC framework. They also note that certain fine-grained categories suffer from low F1 scores. As a direction for future work, the authors propose enhancing the performance of FTBC by incorporating external knowledge bases or gazetteer lists. This could help in better disambiguating entities and improving performance on less frequent or more specific entity types.

Summary updated

BibTeX
@inproceedings{chatterjee2023lsjsp,
  author = {Shilpa Chatterjee and Leo Evenss and Pramit Bhattacharyya and Joydeep Mondal},
  title = {{LSJSP at SemEval-2023 Task 2: FTBC: A FastText based framework with pre-trained BERT for NER}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure