Prompt-based Zero-shot Relation Classification with Semantic Knowledge Augmentation

Jiaying Gong, Hoda EldardiryarXiv 2021 · arXiv · OpenReview

TL;DR

This paper introduces ZS-SKA, a novel prompt-based approach for zero-shot relation extraction that tackles the challenge of identifying relationships not seen during training. By augmenting training data with semantically enriched examples and leveraging external knowledge graphs to construct informative prompts, ZS-SKA significantly outperforms existing methods on benchmark datasets.

What Problem Does This Paper Address?

Relation Triplet Extraction (RTE) is the task of identifying entities and their relationships within text, often represented as <head entity, relation, tail entity>. A major hurdle in RTE is zero-shot learning, where the model must identify relations it has never encountered during training. Traditional supervised methods require labeled data for all possible relations, which is impractical in dynamic, open-world scenarios where new relations constantly emerge. Existing zero-shot approaches often rely on question-answering models or relation descriptions, but they tend to overlook the crucial semantic connections between known (seen) and unknown (unseen) relations. This paper aims to bridge this gap by developing a model that can effectively learn and leverage semantic knowledge to recognize these unseen relations.

Methods

The proposed ZS-SKA model employs a multi-pronged strategy:

  • Data Augmentation: To address the lack of labeled data for unseen relations, ZS-SKA generates synthetic training instances. It uses a novel word-level analogy-based sentence translation rule to transform existing examples of seen relations into new examples for unseen relations. This augmentation is carefully performed to ensure semantic consistency.
  • Prompt-based Semantic Knowledge Augmentation: ZS-SKA designs prompts that integrate semantic knowledge from an external knowledge graph (ConceptNet). Instead of directly using relation names, it constructs "virtual label words" by weighting relevant terms from the knowledge graph. This approach allows the model to infer features of unseen relations by understanding their connections to known concepts.
  • Prototypical Networks: The model learns representations for both seen and unseen relations using the augmented data and prompts. It then employs prototypical networks to compute a "prototype" (an average representation) for each relation. The similarity between a query instance's representation and these prototypes is used to predict unseen relations.
  • Name Entity Extractor: A pre-trained NER model is used to identify entities, and their types are filtered based on the predicted relation's super-class information from the prompt template to construct the final relation triplets.

Datasets

The proposed ZS-SKA model was evaluated on three public datasets:

  • FewRel: A balanced dataset with 80 relation types.
  • Wiki-ZSL: A subset of Wiki-KB, designed for zero-shot learning.
  • NYT: A large, unbalanced, and noisy dataset derived from aligning Freebase relations with the New York Times.

Key Insights and Ideas

  • Bridging the Semantic Gap: The core innovation is leveraging external knowledge graphs not just for relation descriptions but to build nuanced "virtual labels" within prompts. This allows the model to understand the semantic relationships between seen and unseen relations, a critical missing piece in prior work.
  • Data Augmentation with Semantic Constraints: The word-level analogy-based translation rule is crucial for generating high-quality synthetic data. The constraint that the super-class of the relation and entities must match between seen and unseen examples ensures that the augmented data is semantically meaningful.
  • Robustness through Prompts: Using weighted virtual labels in prompts, derived from knowledge graphs, makes the model more robust, especially in scenarios with a high proportion of unseen relations. This is more effective than simply using top-frequency words or actual label names.
  • Prototypical Networks for Unseen Relations: Employing prototypical networks provides a principled way to learn representations for unseen relations without requiring explicit training data for them, by comparing them to learned prototypes.

Key Results

ZS-SKA demonstrated strong performance across the board:

  • Superior Performance: ZS-SKA consistently outperformed state-of-the-art methods on all three datasets for zero-shot relation triplet extraction, particularly in terms of F1-score.
  • Robustness: The model showed remarkable robustness, especially on the noisy NYT dataset and in ablation studies where the percentage of unseen relations was increased.
  • High Precision: ZS-SKA achieved a notably high precision, which is crucial for applications like knowledge graph construction where data quality is paramount.
  • Efficiency: The data augmentation and NER modules do not require additional training, making the approach more efficient.

Limitations and Future Work

  • Data Augmentation Limitations: The current data augmentation is word-level. The authors suggest exploring more advanced generation models for richer augmentation.
  • Knowledge Graph Dependency: The prompt design relies on a fixed knowledge graph. If an unseen relation's label is an out-of-vocabulary word not present in the graph, the model might struggle. Future work could involve using label descriptions or generative models like GPT-2 to handle such cases.
  • Environmental Impact: The paper acknowledges the computational cost of training, noting the carbon footprint associated with GPU usage.
  • Future Directions: The authors propose exploring different instance generation techniques, more sophisticated virtual label construction methods, and leveraging other state-of-the-art data augmentation methods.

Summary updated

Abstract

In relation triplet extraction (RTE), recognizing unseen (new) relations for which there are no training instances is a challenging task. Efforts have been made to recognize unseen relations based on question-answering models or relation descriptions. However, these approaches miss the semantic information about connections between seen and unseen relations. In this paper, We propose a prompt-based model with semantic knowledge augmentation (ZS-SKA) to recognize unseen relations under the zero-shot setting. We present a new word-level analogy-based sentence translation rule and generate augmented instances with unseen relations from instances with seen relations using that new rule. We design prompts with weighted virtual label construction based on an external knowledge graph to integrate semantic knowledge information learned from seen relations. Instead of using the actual label sets in the prompt template, we construct weighted virtual label words. We learn the representations of both seen and unseen relations with augmented instances and prompts. We then calculate the distance between the generated representations using prototypical networks to predict unseen relations. Extensive experiments conducted on three public datasets FewRel, Wiki-ZSL, and NYT, show that ZS-SKA outperforms state-of-the-art methods under the zero-shot scenarios. Our experimental results also demonstrate the effectiveness and robustness of ZS-SKA.
BibTeX
@misc{gong2021prompt,
  author = {Jiaying Gong and Hoda Eldardiry},
  title = {{Prompt-based Zero-shot Relation Classification with Semantic Knowledge Augmentation}},
  year = {2021},
  eprint = {2112.04539},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.04539},
}
Figure