Neural Snowball for Few-Shot Relation Learning

Tianyu Gao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun

Abstract

Knowledge Graphs (KGs) are essential for modern information retrieval but suffer from a "data hunger" problem when dealing with emerging, niche relations. Traditional Relation Extraction (RE) models require extensive labeled data, which is often unavailable for new relations. We present Neural Snowball, a framework that combines the iterative nature of bootstrapping with deep Relational Siamese Networks (RSN). By learning a universal metric of relational similarity from existing data, Neural Snowball can autonomously "grow" a training set from unlabeled text starting from just a few seed examples. Our experiments show that this approach nearly doubles the performance of standard fine-tuning in 5-shot scenarios.

Introduction

The rapid growth of information necessitates the constant addition of new relations to Knowledge Graphs. However, the long-tail nature of these relations means that for many, only a handful of labeled instances exist. This "few-shot" scenario is a significant challenge for supervised learning. While bootstrapping techniques—which use seeds to find more examples—have been used in the past, they often rely on rigid, hand-crafted patterns that fail to capture the linguistic diversity of natural language.

Overview of the Neural Snowball framework, illustrating the transfer of knowledge from existing relations to new ones using unlabeled corpora.

Neural Snowball addresses this by transferring "similarity logic" rather than just features. It uses prior knowledge from well-documented relations to identify when two sentences express the same underlying relationship, even if that relationship is entirely new to the model.

Methodology: The Snowball Loop

The core of the framework is an iterative process driven by two neural components: the Relational Siamese Network (RSN) and a set of binary relation classifiers.

Relational Siamese Networks (RSN)

The RSN is pre-trained on large-scale datasets (like FewRel) to learn a distance metric. Instead of classifying a sentence into a specific category, it learns to determine if two sentences express the same relation. This allows the RSN to act as a high-precision gatekeeper during the data collection process.

Iterative Expansion

The "snowball" effect is achieved through two alternating phases:

  1. Entity Expansion: Starting with seed sentences, the model identifies other sentences in an unlabeled corpus (e.g., Wikipedia) that contain the same entity pairs. The RSN filters these candidates to ensure they actually express the target relation.
  2. Relation Expansion: A binary classifier is trained on the current set of validated instances. This classifier then searches for new entity pairs that likely share the same relation. The RSN again validates these new discoveries before they are added to the training pool.
The detailed workflow of the Neural Snowball process, showing how seeds are expanded into candidate sets and filtered by the Siamese network (S) and Relation Classifier (C).

Experimental Results

We evaluated Neural Snowball using the FewRel benchmark and a Wikipedia unlabeled corpus containing approximately 900,000 instances. We tested the framework using both CNN and BERT architectures to ensure generalizability.

In a 5-shot setup (only 5 seed examples), Neural Snowball with a BERT encoder achieved an F1-score of 47.26. This is a dramatic improvement over simple fine-tuning (25.10) and traditional bootstrapping methods like BREDS (17.58). As the number of seeds increased to 15, the F1-score climbed to 72.06, demonstrating the framework's ability to effectively leverage even small amounts of additional supervision.

Discussion and Limitations

A key insight from our study is the "comfort zone" effect. While the RSN is excellent at maintaining high precision (often over 82% for top-5 selections), it tends to favor linguistic patterns that are semantically similar to the initial seeds. This can limit the "recall" or diversity of the discovered instances. Future work will focus on developing adaptive RSNs that can update their similarity metrics during the snowball process to better explore diverse linguistic expressions.

Conclusion

Neural Snowball represents a significant step forward for open-ended relation learning. By combining the strengths of deep metric learning with iterative bootstrapping, it provides a scalable and accurate way to populate Knowledge Graphs with niche relations, reducing the reliance on massive human-annotated datasets.

Summary updated

Abstract

Knowledge graphs typically undergo open-ended growth of new relations. This cannot be well handled by relation extraction that focuses on pre-defined relations with sufficient training data. To address new relations with few-shot instances, we propose a novel bootstrapping approach, Neural Snowball, to learn new relations by transferring semantic knowledge about existing relations. More specifically, we use Relational Siamese Networks (RSN) to learn the metric of relational similarities between instances based on existing relations and their labeled data. Afterwards, given a new relation and its few-shot instances, we use RSN to accumulate reliable instances from unlabeled corpora; these instances are used to train a relation classifier, which can further identify new facts of the new relation. The process is conducted iteratively like a snowball. Experiments show that our model can gather high-quality instances for better few-shot relation learning and achieves significant improvement compared to baselines. Codes and datasets are released on https://github.com/thunlp/Neural-Snowball.
BibTeX
@inproceedings{gao2020neural,
  author = {Tianyu Gao and Xu Han and Ruobing Xie and Zhiyuan Liu and Fen Lin and Leyu Lin and Maosong Sun},
  title = {{Neural Snowball for Few-Shot Relation Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6281},
}
Figure