ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning

Chih-Yao Chen, Cheng-Te LiNAACL 2021 · DOI

Abstract

Relation Extraction (RE) is a fundamental task in information extraction, yet traditional models struggle with the "closed-world" assumption, failing to recognize relations not seen during training. This paper introduces ZS-BERT, a framework designed for Zero-Shot Relation Extraction (ZSRE). By mapping sentences and textual relation descriptions into a shared semantic embedding space, ZS-BERT treats relation extraction as a nearest-neighbor search problem. Experimental results on Wiki-ZSL and FewRel datasets demonstrate that ZS-BERT significantly outperforms existing state-of-the-art methods, achieving over a 13% improvement in F1 score and showing robust scalability to an increasing number of unseen relations.

Introduction

In the dynamic landscape of real-world knowledge bases, new relationship types emerge constantly. Traditional supervised RE models are limited by their fixed label sets, requiring extensive retraining and manual labeling for every new relation. While Zero-Shot Learning (ZSL) offers a solution, previous ZSRE approaches often relied on unscalable manual template engineering or computationally expensive text entailment tasks.

Zero-Shot Relation Extraction Concept: The model aligns sentences and relation descriptions in an embedding space during training to enable the identification of unseen relations at test time.

ZS-BERT addresses these challenges by leveraging publicly available textual descriptions of relations (e.g., from Wikidata) as semantic anchors, enabling the model to generalize to unseen relations without additional human intervention.

Methods

The core of ZS-BERT is a multi-task learning framework that aligns sentence representations with relation descriptions in a common semantic space.

ZS-BERT Architecture: Dual encoders map input sentences and relation descriptions into a shared embedding space for prediction.

The architecture consists of two primary components:

  1. Dual Encoders:
  2. Sentence Encoder: A BERT-based model processes the input sentence, capturing global context via the [CLS] token and specific entity representations. These are concatenated to form a robust sentence embedding.
  3. Description Encoder: A Sentence-BERT (S-BERT) model transforms textual relation descriptions (e.g., "main creator(s) of a written work") into fixed-length attribute vectors.
  4. Learning Objectives: The model is trained using a multi-task approach. A Ranking Loss (Alignment) minimizes the distance between a sentence embedding and its corresponding relation's attribute vector while maximizing the distance to incorrect relations using a margin-based approach. Simultaneously, a Cross-Entropy Loss (Classification) is applied to seen relations to ensure the BERT encoder learns discriminative features.

The Zero-Shot Leap

At inference time, the model encounters sentences belonging to "unseen" relation classes. It generates a sentence embedding and identifies the most likely relation by performing a nearest-neighbor search among the attribute vectors of all candidate unseen relations. This approach allows the model to handle an arbitrary number of new relations as long as a textual description is available.

Results and Discussion

Evaluation on the Wiki-ZSL and FewRel benchmarks shows that ZS-BERT achieves substantial performance gains. It outperforms previous baselines like CIM and ESIM by at least 13.54% in F1 score. Notably, ZS-BERT exhibits superior scalability; as the number of unseen relations increases, its performance degrades much more gracefully than traditional supervised models.

Furthermore, the model demonstrates impressive few-shot adaptability. With as little as 2% of labeled data for previously unseen relations, ZS-BERT's F1 score jumps to 80%, highlighting its potential for rapid domain adaptation.

Conclusion

ZS-BERT presents a practical and effective approach to Zero-Shot Relation Extraction by utilizing "free" supervision from existing relation descriptions. By framing RE as a representation learning and retrieval task, it overcomes the scalability issues of previous methods. Future work will focus on mitigating entity confusion caused by global sentence context and addressing noise in distantly supervised training data to further push the boundaries of zero-shot performance.

Summary updated

BibTeX
@inproceedings{chen2021zs,
  author = {Chih-Yao Chen and Cheng-Te Li},
  title = {{ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.272},
}
Figure