PAI at SemEval-2023 Task 2: A Universal System for Named Entity Recognition with External Entity Information

Long Ma, Kai Lü, Tianbo Che, Hailong Huang, Weiguo Gao, Xuan Li

Abstract

The MultiCoNER II task aims to detect complex, ambiguous, and fine-grained named entities in low-context situations and noisy scenarios, such as those containing spelling mistakes and typos. These conditions pose significant challenges due to the scarcity of contextual information and the high granularity of entities (up to 33 classes). To address these issues, the PAI team proposes a universal Named Entity Recognition (NER) system that integrates external entity information from knowledge bases like WikiData. By concatenating retrieved entity properties with input sentences and utilizing a specialized Transformer-based model, the system achieved 2 first places, 4 second places, and 1 third place across 13 tracks.

The Challenge: Fine-Grained NER in Low-Context Scenarios

Named Entity Recognition (NER) is a fundamental NLP task, but the MultiCoNER II challenge introduces hurdles that break traditional models. Standard models often fail when encountering:

  1. Low Context: Short snippets like search queries provide few clues for disambiguation.
  2. High Granularity: Models must distinguish between 33 fine-grained classes (e.g., "Scientist" vs. "Artist").
  3. Noisy Data: Typos and spelling errors interfere with standard dictionary lookups.

As illustrated below, without external knowledge, a model cannot reliably determine if an entity like "Bryan Talbot" is a politician, scientist, or artist. External knowledge provides the necessary signal to make an accurate prediction.

The benefit of external knowledge for disambiguating entities in low-context sentences.

Methodology: Knowledge-Augmented NER

The PAI system functions through a three-stage pipeline designed to bridge the context gap:

1. Knowledge Retrieval Module

For any input sentence, the system performs string matching against a massive knowledge base derived from WikiData (90M+ entities). It retrieves specific properties such as alias, description, instanceof, subclassof, and occupation.

2. Context Augmentation

The retrieved information is formatted into a text string and concatenated with the original sentence. For example, the name "Lanny Ross" is augmented with properties like "occupation: singer-songwriter."

3. Entity-Aware Attention

This is the core innovation of the system. Rather than using standard self-attention, the authors designed a custom entity-aware attention mask. This mask ensures the model specifically models the relationship between an entity in the sentence and its corresponding external context, preventing the model from being distracted by irrelevant retrieved data.

The PAI system architecture featuring entity-aware attention and external knowledge retrieval.

Key Insights and Experimental Results

The impact of external knowledge was transformative. The PAI system secured top rankings across nearly all 13 competition tracks. Quantitatively, the inclusion of external properties led to a 12.2 point jump in average F1 score (from 71.3 to 83.5) compared to a baseline without external information.

  • Property Specificity: The authors found that different properties served different entity types. The occupation property was critical for identifying "Person" sub-types, while subclassof provided the strongest signal for "Product" categories.
  • Global Knowledge Coverage: The WikiData-based approach achieved over 95% entity coverage even for low-resource languages like Bangla and Hindi, proving that global knowledge bases are viable across diverse linguistic contexts.
  • Attention Efficacy: The entity-aware attention mechanism provided a measurable boost of up to 1.0 F1 point by helping the model focus on the most relevant external signals.

Conclusion and Future Directions

The PAI system demonstrates that external knowledge is a powerful tool for overcoming the limitations of low-context NER. While highly effective, the authors noted that current string-matching retrieval is vulnerable to noise. Future work will focus on integrating fuzzy matching (e.g., via ElasticSearch) to better handle typos and exploring more efficient ways to encode long property strings within the sequence length limits of Transformer models.

Summary updated

Abstract

The MultiCoNER II task aims to detect complex, ambiguous, and fine-grained named entities in low-context situations and noisy scenarios like the presence of spelling mistakes and typos for multiple languages. The task poses significant challenges due to the scarcity of contextual information, the high granularity of the entities(up to 33 classes), and the interference of noisy data. To address these issues, our team {\bf PAI} proposes a universal Named Entity Recognition (NER) system that integrates external entity information to improve performance. Specifically, our system retrieves entities with properties from the knowledge base (i.e. Wikipedia) for a given text, then concatenates entity information with the input sentence and feeds it into Transformer-based models. Finally, our system wins 2 first places, 4 second places, and 1 third place out of 13 tracks. The code is publicly available at \url{https://github.com/diqiuzhuanzhuan/semeval-2023}.
BibTeX
@misc{ma2023pai,
  author = {Long Ma and Kai Lü and Tianbo Che and Hailong Huang and Weiguo Gao and Xuan Li},
  title = {{PAI at SemEval-2023 Task 2: A Universal System for Named Entity Recognition with External Entity Information}},
  year = {2023},
  eprint = {2305.06099},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.06099},
}
Figure