Zshot: An Open-source Framework for Zero-Shot Named Entity Recognition and Relation Extraction

Gabriele Picco, Marcos Martínez Galindo, Alberto Purpura, Leopold Fuchs, Vanessa López, Thanh Lam Hoang

Introduction

In traditional Natural Language Processing (NLP), training models to recognize specific entities (e.g., "Protein Name") or relations (e.g., "Works For") typically requires thousands of manually labeled examples. This data-hungry approach is expensive, time-consuming, and often prohibitive for niche domains like medicine or law.

Zero-Shot Learning (ZSL) offers a promising alternative by allowing models to identify new classes using auxiliary information, such as natural language descriptions. However, the ZSL field has been fragmented, with researchers using varied assumptions, inconsistent evaluation metrics, and incompatible codebases. This lack of standardization hinders both industrial adoption and fair academic comparison.

The Zshot Framework

To address these challenges, the authors introduce Zshot, an open-source framework designed to unify and standardize the ZSL pipeline. Zshot is built as an extension for the spaCy library, enabling seamless integration into existing NLP workflows with minimal code changes.

The Zshot modular pipeline architecture, illustrating the flow from text input to predictions and evaluation.

The framework decomposes the ZSL process into three modular components:

  1. Mention Detection: Identifying potential entity spans within the text (e.g., using Flair or spaCy).
  2. Entity Linking/Classification: Assigning labels to detected spans based on provided descriptions. Zshot supports state-of-the-art models such as SMXM, BLINK, GENRE, and TARS.
  3. Relation Extraction: Determining the connections between identified entities (e.g., using ZS-BERT).

Beyond its modularity, Zshot introduces features like ensembling, which allows users to combine multiple models or descriptions to improve stability and accuracy. It also extends spaCy’s displaCy utility to visualize zero-shot relations and entities effectively.

The Power of Descriptions

The core innovation of ZSL is the use of "Class Definitions" as a bridge to unseen categories. By providing a descriptive prompt, the model can leverage its pre-trained knowledge to disambiguate terms. For instance, a model can distinguish between "Apple" the company and "apple" the fruit based solely on their respective definitions.

Example of entity disambiguation where natural language descriptions allow the model to distinguish between a company and a fruit.

Zshot further enhances this by allowing multiple descriptions for a single entity, using a voting mechanism to mitigate the sensitivity of ZSL models to specific phrasing.

Evaluation and Results

To ensure reproducibility, Zshot includes standardized interfaces for key ZSL datasets, including OntoNotesZS (NER), MedMentionsZS (biomedical NER), and FewRel (relation extraction).

In benchmarking tests on OntoNotesZS, the authors compared the SMXM and TARS models:

  • Performance: TARS demonstrated superior results, achieving a micro-F1 of 40.47% compared to SMXM's 30.76%.
  • Efficiency: TARS was significantly more performant, processing 0.69 samples per second, whereas SMXM managed 0.23 samples per second on identical hardware.

Limitations and Future Work

While Zshot represents a significant step forward, several challenges remain. The current framework is primarily optimized for English, and the underlying models are susceptible to social biases present in their training data (e.g., gender-biased occupational labeling). Furthermore, while "zero-shot," performance is still correlated with how well the base language model represents the target domain. Future development will focus on expanding language support, improving computational efficiency, and integrating a wider array of models.

Summary updated

BibTeX
@inproceedings{picco2023zshot,
  author = {Gabriele Picco and Marcos Martínez Galindo and Alberto Purpura and Leopold Fuchs and Vanessa López and Thanh Lam Hoang},
  title = {{Zshot: An Open-source Framework for Zero-Shot Named Entity Recognition and Relation Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-demo.34},
}
Figure