PAI at SemEval-2022 Task 11: Name Entity Recognition with Contextualized Entity Representations and Robust Loss Functions

Long Ma, Xiaorong Jian, Xuan LiProceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022) 2022 · DOI

TL;DR

The researchers from Pingan Life Insurance developed Dictionary-fused BERT, a system that clinches a top spot in complex Named Entity Recognition (NER) by blending external knowledge with robust training techniques. By integrating entity dictionaries directly into the model architecture and using a specialized loss function to distinguish between easily confused categories, they achieved a post-evaluation F1 score of 0.914—surpassing the official first-place leaderboard score.

What Problem Does This Paper Address?

Standard Named Entity Recognition (NER) models are great at identifying "easy" entities like common person names or well-known locations. However, they often stumble on complex entities—like the titles of movies, books, or software—especially in short, low-context snippets like search queries or code-mixed text (e.g., a query containing both English and Spanish).

The core challenge is that these complex entities don't follow simple noun patterns and often overlap with common vocabulary. Traditional models rely heavily on memorizing training data, which fails when they encounter "emerging" or "long-tail" entities that weren't in the original dataset.

Methods

The authors propose a three-pronged approach to make NER more "knowledge-aware" and resilient:

  1. Dictionary-fused BERT: Building on the LUKE architecture, this method treats entities from an external dictionary as separate tokens. When a sentence is processed, the model performs "Dictionary Entity Matching." It identifies potential entities in the text and feeds them into the Transformer alongside the original words, allowing the model to create contextualized representations for both words and entities simultaneously.
  2. Robust Loss Function ($L_{logit}$): To prevent overfitting and help the model distinguish between similar categories (like "Creative Work" vs. "Product"), the authors designed a loss function that leverages a logit matrix. It uses Kullback-Leibler (KL) divergence to penalize the model if it gets too "confused" between specific tag pairs that are historically difficult to separate.
  3. Auxiliary Binary Task: The model doesn't just predict the entity type; it also performs a simpler "on-top" binary classification to decide if a token is part of any entity mention or not. This auxiliary task acts as a "hint," making the primary, more complex NER task easier for the model to learn and improving overall recall.

Datasets

  • MultiCoNER: The primary dataset for SemEval-2022 Task 11, featuring 15,300 training samples and a massive test set of over 217,000 samples across six entity types (Person, Location, Corporation, Groups, Product, and Creative-Work).
  • LUKE Entity Dictionary: A collection of 500,000 entities retrieved from English Wikipedia.
  • GEMNET Dictionary: Used in the post-evaluation stage to further boost performance.

Key Insights and Ideas

  • Knowledge over Memorization: The most significant "aha" moment is that integrating an external dictionary allows the model to handle "unseen" entities without needing to retrain the entire model. It shifts the burden from the model's weights to an external, updatable knowledge base.
  • The "Confusion" Fix: The authors noticed that certain categories (like Corporations vs. Groups) are frequently misclassified. By specifically targeting these "confusable pairs" in the loss function using KL divergence, they forced the model to create more distinct boundaries between these labels.
  • Multi-Task Synergy: Adding the simple binary "is this a mention?" task provided a 0.4% improvement in F1 score. While small, it demonstrates that simplifying the search space for the model helps it focus on the harder task of categorization.

Key Results

  • Base Model Selection: After testing various architectures (RoBERTa, DistilBERT, etc.), BERT-Whole-Word-Masking (Large) emerged as the strongest backbone, achieving a baseline F1 of 0.883.
  • Ablation Success: Each component added clear value. Adding the entity dictionary boosted the score to 0.90, the auxiliary task pushed it to 0.904, and the robust logit loss brought it to 0.913 on the development set.
  • State-of-the-Art Performance: In the post-evaluation phase, by simply swapping the LUKE dictionary for the GEMNET dictionary, the system reached an F1 of 0.914, which was higher than the official 1st place score on the competition leaderboard.

Limitations and Future Work

The authors note that while their method is effective, the way entity spans are currently encoded is somewhat implicit—relying on a special $ separator token. They suggest that future research should explore more explicit ways to encode the boundaries (spans) of entities to further improve accuracy. Additionally, they aim to investigate other methods for infusing entity information into Transformer-based models beyond the current dictionary-matching approach.

Summary updated

Abstract

This paper describes our system used in the SemEval-2022 Task 11 Multilingual Complex Named Entity Recognition, achieving 3rd for track 1 on the leaderboard. We propose Dictionary-fused BERT, a flexible approach for entity dictionaries integration. The main ideas of our systems are: 1) integrating external knowledge (an entity dictionary) into pretrained models to obtain contextualized word and entity representations 2) designing a robust loss function leveraging a logit matrix 3) adding an auxiliary task, which is an on-top binary classification to decide whether the token is a mention word or not, makes the main task easier to learn. It is worth noting that our system achieves an F1 of 0.914 in the postevaluation stage by updating the entity dictionary to the one of
BibTeX
@inproceedings{ma2022pai,
  author = {Long Ma and Xiaorong Jian and Xuan Li},
  title = {{PAI at SemEval-2022 Task 11: Name Entity Recognition with Contextualized Entity Representations and Robust Loss Functions}},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.229},
}
Figure