Leveraging Type Descriptions for Zero-shot Named Entity Recognition and Classification

Rami Aly, Andreas Vlachos, Ryan McDonaldACL 2021 · DOI

TL;DR

This paper introduces the first approach to zero-shot Named Entity Recognition and Classification (NERC), enabling models to identify and categorize entities without prior training examples for those specific entity types. By leveraging textual descriptions of entity types, the proposed SMXM model significantly outperforms existing baselines, demonstrating the power of descriptive knowledge in tackling data scarcity.

What Problem Does This Paper Address?

Named Entity Recognition and Classification (NERC) is a fundamental task in Natural Language Processing (NLP) that involves identifying and categorizing entities (like people, organizations, or locations) within text. A major hurdle in real-world NERC applications is the lack of annotated data for all possible entity types. This scarcity necessitates "zero-shot" learning, where models must generalize to unseen entity types based on knowledge learned from observed types. While zero-shot approaches exist for related tasks like Named Entity Typing (NET), applying this to the full NERC task (which includes both recognition and classification) has remained an open challenge. A key difficulty is defining the "not-an-entity" class, as tokens labeled as non-entities during training might actually belong to unseen entity types during testing.

Methods

The core contribution is a novel architecture called SMXM (Sequential Multiclass Cross-attention Model). SMXM leverages cross-attention mechanisms, powered by transformer encoders (like BERT), to model the semantic affinity between a sentence and the textual descriptions of entity types.

Key aspects of the method include:

  • Cross-Attention Encoder (X-ENC): This component uses a transformer to generate representations for each token in the sentence by attending to the entity type descriptions.
  • Modelling the Negative Class: The paper proposes and evaluates three strategies for handling the "not-an-entity" class:
  • Description-based encoding: Using a textual description for the negative class.
  • Independent encoding: Directly modeling the negative class without descriptions.
  • Class-aware encoding: Combining representations from all positive classes to infer the negative class. The authors found class-aware encoding to be the most effective.
  • Entity Masking: During training, a regularization technique inspired by BERT's masked language modeling is used. Entire entities are masked, forcing the model to learn context-aware representations and preventing overfitting to specific lexical items.

Datasets

The authors adapt two existing datasets for zero-shot NERC evaluation:

  • OntoNotes: A general-purpose NERC benchmark.
  • MedMentions: A domain-specific biomedical dataset.

They create zero-shot splits by selecting the rarest entity classes for training and using the remaining classes for development and testing, ensuring disjoint sets. They also identify and exclude "trivial" classes (e.g., PERCENT) that are easily recognizable with simple rules, focusing on more challenging scenarios.

Key Insights and Ideas

  • Descriptions are Crucial: The paper strongly advocates for the use of rich textual descriptions (like annotation guidelines) for entity types, showing they are more effective than just class names or Wikipedia snippets for transferring knowledge.
  • The Negative Class is Tricky: Accurately modeling the "not-an-entity" class is critical for zero-shot NERC, and the proposed class-aware encoding approach proves superior to simpler methods.
  • Entity Masking Aids Generalization: The entity masking strategy during training helps prevent overfitting and leads to better generalization to unseen entity types, as evidenced by improved validation F1 scores.
  • Larger Pre-trained Models Matter: Using BERT-Large significantly outperforms models based on BERT-Base, highlighting the benefit of larger-scale pre-training for domain-specific zero-shot tasks.

Key Results

The proposed SMXM model achieves state-of-the-art results on the zero-shot NERC task:

  • OntoNotes-ZS: Macro-averaged F1 score of 0.45 (token-level).
  • MedMentions-ZS: Macro-averaged F1 score of 0.38 (token-level).

SMXM outperforms baseline models like Binary Entailment Model (BEM) and Machine Reading Comprehension for NERC (MRC). The analysis of entity descriptions shows that annotation guidelines for OntoNotes and Metathesaurus descriptions for MedMentions yield the best performance.

Limitations and Future Work

  • Description Quality: The performance is highly dependent on the quality and clarity of entity type descriptions. Vague, noisy, or negated descriptions can negatively impact performance.
  • Non-entity Class Complexity: While class-aware encoding is effective, the inherent difficulty of defining the negative class in zero-shot settings remains a challenge.
  • Future Work: The authors suggest incorporating dependencies between predicted labels and exploring more sophisticated ways to leverage semi-structured knowledge from descriptions. They also note that modifications to UMLS descriptions showed promise, indicating room for further improvement in description engineering.

Summary updated

BibTeX
@inproceedings{aly2021leveraging,
  author = {Rami Aly and Andreas Vlachos and Ryan McDonald},
  title = {{Leveraging Type Descriptions for Zero-shot Named Entity Recognition and Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.120},
}
Figure