FewRel 2.0: Towards More Challenging Few-Shot Relation Classification

Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

TL;DR

FewRel 2.0 upgrades the popular few-shot relation classification benchmark by introducing two grueling real-world hurdles: domain adaptation (moving from Wikipedia to biomedical text) and "none-of-the-above" (NOTA) detection. The authors demonstrate that state-of-the-art models fail dramatically under these conditions, and they introduce a new binary-matching baseline, BERT-PAIR, to help close the gap.

What Problem Does This Paper Address?

Few-shot learning has made massive strides in Natural Language Processing (NLP), particularly in relation classification (RC)—the task of identifying relationships between entities in text (e.g., recognizing that "Mark Twain was born in 1835" expresses the relation date_of_birth).

However, existing benchmarks like FewRel 1.0 operate under two unrealistic assumptions:

  1. Domain Homogeneity: The training and testing data come from the same domain (typically Wikipedia). In reality, we want to train a model on data-rich domains and deploy it on data-sparse domains like medicine or finance.
  2. Closed-World Assumption: Models assume that every query sentence must fit into one of the pre-defined target relations. In the wild, most sentences express completely unrelated information, requiring a "none-of-the-above" (NOTA) option.

FewRel 2.0 addresses these gaps by introducing a new benchmark specifically designed to test Few-Shot Domain Adaptation (DA) and Few-Shot NOTA Detection.

Figure 1: The comparison between the best results of the current models on FewRel, few-shot DA and few- shot NOTA. From the figures we can see that even the state-of-the-art models struggle on the new tasks.

Methods

To tackle these new challenges, the authors evaluate standard few-shot architectures (like Prototypical Networks and Graph Neural Networks) alongside two targeted approaches:

  • For Domain Adaptation: They implement an adversarial training framework. A sentence encoder extracts features, while a domain discriminator tries to guess whether the text came from Wikipedia or the biomedical domain. By optimizing this min-max game, the encoder learns to produce domain-invariant features.
  • For NOTA Detection (BERT-PAIR): Instead of treating NOTA as a simple $(N+1)$-th class—which fails because NOTA instances are scattered randomly across the semantic space—the authors propose a binary matching approach. BERT-PAIR pairs the query instance with every single supporting instance in the few-shot context. It feeds these pairs into a BERT sequence classification model to calculate a similarity score ("same relation" vs. "different relation"). The NOTA score is then determined by how poorly the query matches all available target relations.

Datasets

The paper introduces a brand-new test set to facilitate these tasks:

  • Training Set: The original FewRel 1.0 dataset, built from Wikipedia corpus and Wikidata knowledge bases.
  • Domain Adaptation Test Set: A newly annotated biomedical dataset created by aligning PubMed abstracts with the UMLS (Unified Medical Language System) knowledge base. It consists of 25 distinct biomedical relations with 100 manually verified instances each.
  • Validation Set: The SemEval-2010 Task 8 dataset, providing an intermediate domain shift.

Key Insights and Ideas

  • The Domain Shift Penalty is Severe: Moving from general Wikipedia text to highly specialized biomedical jargon causes model performance to plummet. The syntax, vocabulary, and relation types are vastly different, exposing the fragility of current meta-learning models.
  • Standard NOTA Strategies Fail: Simply sampling random negative instances during training to act as a "NOTA class" doesn't work well. Because NOTA represents an infinite semantic space of "everything else," standard classification layers cannot draw clean decision boundaries around it.
  • Framing Classification as Matching Helps: BERT-PAIR's success stems from transforming an $N$-way classification problem into a series of 1-to-1 verification steps. This makes the model inherently more robust to open-world scenarios where the correct answer might be "none of them."

Key Results

The experimental results highlight just how difficult FewRel 2.0 is:

Few-Shot Domain Adaptation

When tested on the new biomedical domain (On 2.0), standard models experienced massive performance drops. For example, Proto (BERT) dropped from 80.68% accuracy on Wikipedia to just 40.12% on the biomedical test set in a 5-Way 1-Shot setting. While adversarial training (Proto-ADV) helped slightly, the new BERT-PAIR model achieved the highest adaptation score at 56.25%.

Few-Shot NOTA Detection

As the percentage of NOTA queries increases, standard models break down if they aren't equipped to handle them. However, BERT-PAIR proved remarkably stable. In a 5-Way 1-Shot setup with a 50% NOTA rate, BERT-PAIR maintained an accuracy of 80.31%, vastly outperforming standard Prototypical Networks.

Figure 2: 5-way K-shot results under different NOTA rates. Models with * simply ignore the NOTA setting and assume all queries can be classified as one of the N relations.

Limitations and Future Work

Despite BERT-PAIR's improvements, a massive performance gap remains between in-domain performance and cross-domain/NOTA performance.

Interestingly, the authors noted that adding just 1,000 labeled instances from the target biomedical domain into the training set boosted BERT-PAIR's performance by 13 to 16 percentage points. This indicates that purely unsupervised domain adaptation is still incredibly difficult, and future work must focus on developing better unsupervised domain-invariant representations or more effective ways to utilize minimal target-domain data.

Summary updated

Abstract

We present FewRel 2.0, a more challenging task to investigate two aspects of few-shot relation classification models: (1) Can they adapt to a new domain with only a handful of instances? (2) Can they detect none-of-the-above (NOTA) relations? To construct FewRel 2.0, we build upon the FewRel dataset (Han et al., 2018) by adding a new test set in a quite different domain, and a NOTA relation choice. With the new dataset and extensive experimental analysis, we found (1) that the state-of-the-art few-shot relation classification models struggle on these two aspects, and (2) that the commonly-used techniques for domain adaptation and NOTA detection still cannot handle the two challenges well. Our research calls for more attention and further efforts to these two real-world issues. All details and resources about the dataset and baselines are released at https: //github.com/thunlp/fewrel.
BibTeX
@inproceedings{gao2019fewrel,
  author = {Tianyu Gao and Xu Han and Hao Zhu and Zhiyuan Liu and Peng Li and Maosong Sun and Jie Zhou},
  title = {{FewRel 2.0: Towards More Challenging Few-Shot Relation Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1649},
}
Figure