SRCB at SemEval-2023 Task 2: A System of Complex Named Entity Recognition with External Knowledge

Yuming Zhang, Hongyu Li, Yongwei Zhang, Shanshan Jiang, Bin DongACL 2023

TL;DR

This paper tackles the challenging task of recognizing complex, ambiguous named entities in short, low-context text snippets across multiple languages. The SRCB team developed a system that augments input text with external knowledge from Wikidata and Wikipedia, significantly improving performance and achieving 3rd place in the Chinese track and 6th place in the English track of SemEval-2023 Task 2.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), aiming to identify and categorize entities (like people, organizations, locations) within text. However, traditional NER systems often struggle with "complex named entities," which can be phrased in various ways and lack sufficient context for clear identification. This is particularly true in tasks like MultiCoNER II, which focuses on fine-grained entity categories in short, low-context settings for multiple languages, and even introduces spelling mistakes to simulate real-world noise. The core challenge is disambiguating these entities when context is scarce, making it difficult for models to determine what an entity is and what category it belongs to.

Methods

The SRCB system employs an external knowledge-based approach, integrating information beyond the immediate text. The system comprises two main modules:

  1. Knowledge Retrieval Module:
    • Mention Detection: A high-recall mention detection model (specifically, a global pointer model) is used to identify potential entity spans in the input text.
  2. External Knowledge Acquisition: For each detected potential entity, the system retrieves three types of external knowledge:
  3. Prompt: Possible labels for the entity, derived from Wikidata.
  4. Description: Concise descriptions of the entity from Wikidata (translated to Chinese for the Chinese track).
  5. Context: The most similar sentence and its surrounding paragraph from Wikipedia.
  6. Knowledge Integration: The original text is concatenated with these three types of external knowledge. Special tokens are used to delineate these different components for the NER model.
  7. Named Entity Recognition (NER) Module:
    • Model Architectures: The system utilizes various NER models, including sequence tagging approaches (Normal, EntLM, LEAR) and a Machine Reading Comprehension (MRC) based approach. Different pre-trained language models (BERT, RoBERTa, Ernie) are explored.
  8. Ensemble Method: To further boost performance, the outputs of multiple trained models are combined using majority voting and a novel random voting strategy.

Datasets

The paper evaluates its system on the MultiCoNER II dataset. This dataset is characterized by:

  • Multilingual: Supporting multiple languages.
  • Complex Entities: Entities that can be composed of any form of language.
  • Fine-grained Categories: 33 specific categories for complex entities.
  • Short and Low-Context Settings: Limiting the amount of surrounding text.
  • Noisy Test Set: Including spelling mistakes and typos to simulate real-world data challenges.

Key Insights and Ideas

  • External Knowledge is Crucial for Ambiguity: The paper strongly reinforces the idea that external knowledge is not just helpful but essential for resolving ambiguity in low-context NER tasks.
  • Wikidata and Wikipedia as Rich Sources: Leveraging structured knowledge from Wikidata (for entity types and descriptions) and unstructured context from Wikipedia proves effective.
  • Global Pointer for High Recall: The choice of a global pointer model for mention detection is critical, prioritizing recall to ensure no potential entities are missed, as missing information is more detrimental than a false positive.
  • Concatenation Strategy Works: A straightforward approach of concatenating the original text with external knowledge, while carefully formatting it, yields significant improvements.
  • Random Voting for Robustness: The random voting ensemble method shows promise, potentially offering better generalization than simple majority voting by mitigating the impact of individual model biases.

Key Results

  • Competitive Performance: The SRCB system achieved 3rd place in the Chinese track and 6th place in the English track of SemEval-2023 Task 2.
  • Significant Improvement: The use of external knowledge led to substantial performance gains compared to baseline models that do not use such information. For instance, on the development set, sequence tagging models trained with external knowledge achieved F1 scores of 88.72 (Chinese) and 77.25 (English), significantly outperforming baselines of 72.94 and 67.42, respectively.
  • Impact of Noise: The system's performance degraded on the noisy test set compared to the clean development set, highlighting the challenge posed by spelling mistakes and typos, especially for the knowledge retrieval module.

Limitations and Future Work

  • Sensitivity to Noise: The knowledge retrieval module struggles with spelling mistakes and typos in the test set, leading to a degradation in the quality of retrieved external knowledge.
  • Limited Fuzzy Matching: Fuzzy matching was only applied to context retrieval, not for Prompt or Description, which could have improved their coverage.
  • Future Directions:
    • Improve knowledge retrieval robustness against noise (spelling mistakes, typos).
  • Explore more sophisticated ways of integrating external knowledge, potentially by modifying model structures.
  • Focus on the importance of contextual information to reduce reliance on external knowledge.

Summary updated

BibTeX
@inproceedings{zhang2023srcb,
  author = {Yuming Zhang and Hongyu Li and Yongwei Zhang and Shanshan Jiang and Bin Dong},
  title = {{SRCB at SemEval-2023 Task 2: A System of Complex Named Entity Recognition with External Knowledge}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure