DSP: Discriminative Soft Prompts for Zero-Shot Entity and Relation Extraction

Bo Lv, Xin Liu, Shaojie Dai, Nayu Liu, Fan Yang, Ping Luo, Yue Yu

Abstract

Zero-shot entity and relation extraction are critical for open-domain information extraction but suffer from the "verbalizer bottleneck" and high computational costs of traditional cloze-style prompting. This paper introduces Discriminative Soft Prompts (DSP), a framework that reframes extraction as a token discrimination task. By employing Discriminative Language Models (DLMs) and a novel soft prompt co-reference strategy, DSP achieves significant gains in both accuracy and inference speed, outperforming state-of-the-art models by up to 7.5% in F1-score while running up to 120 times faster.

Introduction

Extracting information from text—identifying entities (Named Entity Recognition, or NER) and their relationships (Relation Extraction, or RE)—is a cornerstone of Natural Language Processing. In a zero-shot setting, models must identify categories they have never encountered during training, a task that is notoriously difficult.

Current state-of-the-art methods typically use "prompts" to turn extraction into a cloze task (e.g., "Jordan is a [MASK] entity"). However, this approach faces two major hurdles:

  1. The Verbalizer Bottleneck: Mapping complex labels (like "Work of Art") to a single word for the model to predict is difficult and often arbitrary.
  2. The Speed Trap: To identify entities, these models often check every possible text span individually, leading to $O(n^2)$ complexity, which is too slow for real-world applications.

Methodology

DSP addresses these issues through two primary innovations: Discriminative Prompting and Soft Prompt Co-reference.

1. Discriminative Prompting

Instead of using Masked Language Models (MLMs) like BERT, DSP utilizes Discriminative Language Models (DLMs) like ELECTRA. During pre-training, DLMs learn to detect if a token is "original" or "replaced." DSP leverages this by creating a template: "The entity type of [Entity] is [Label]." The model then decides if the [Label] is "original" (correct) or "replaced" (incorrect), eliminating the need for complex verbalizers.

2. Soft Prompt Co-reference

To overcome the $O(n^2)$ speed bottleneck, the authors designed a soft prompt co-reference strategy. Instead of re-processing the entire sentence for every candidate span, they insert special "soft prompt" tokens ($[s]$ for start, $[e]$ for end) into the input. These tokens are trained to "point" to the vector representations of the actual text tokens. This allows the model to identify all entities of a specific type in a single pass.

The DSP architecture for ZSNER and ZSRC using soft prompt co-reference.

Experimental Results

The model was evaluated on four major benchmarks: OntoNotes 5.0 and MedMentions for NER, and Wiki-ZSL and FewRel for RE. The researchers followed a strict zero-shot protocol, testing on categories the model never saw during fine-tuning.

Accuracy and Robustness

DSP outperformed previous state-of-the-art models across the board. It achieved a 7.5% average F1-score improvement on relation extraction tasks. As shown in the results table, DSP maintains high performance even as the number of unseen labels ($m$) increases, demonstrating superior generalization.

Performance comparison of DSP-ZSRC against baselines on Wiki-ZSL and FewRel datasets across varying numbers of unseen labels (m).

Efficiency

The most dramatic result was the leap in inference speed. On the OntoNotes dataset, DSP was 120x faster than standard cloze-style prompts and 416x faster than the baseline DLM approach. This efficiency stems from the soft prompt strategy, which avoids redundant computations for overlapping text spans.

Discussion and Conclusion

The success of DSP suggests that the "token discrimination" task used to train models like ELECTRA is more naturally aligned with the classification nature of entity and relation extraction than BERT's "word guessing" task. By keeping the task format similar to pre-training, DSP also avoids "catastrophic forgetting," retaining more of the model's general linguistic knowledge.

While powerful, the current system uses a pipeline approach (one model for entities, another for relations), which can lead to error propagation. Future work will focus on developing a unified model to handle both tasks simultaneously and further optimizing reasoning complexity.

Reference: Bo Lv, et al. (2023). DSP: Discriminative Soft Prompts for Zero-Shot Entity and Relation Extraction. Proceedings of ACL 2023.

Summary updated

BibTeX
@inproceedings{lv2023dsp,
  author = {Bo Lv and Xin Liu and Shaojie Dai and Nayu Liu and Fan Yang and Ping Luo and Yue Yu},
  title = {{DSP: Discriminative Soft Prompts for Zero-Shot Entity and Relation Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.339},
}
Figure