FewRel 2.0: Towards More Challenging Few-Shot Relation Classification

Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

Abstract

Few-shot relation classification (RC) models have reached a performance plateau on existing benchmarks, often surpassing human-level accuracy on "clean" datasets like FewRel 1.0. However, these benchmarks frequently fail to reflect real-world complexities such as domain shifts and irrelevant queries. This paper introduces FewRel 2.0, a more challenging dataset that incorporates two grueling real-world hurdles: domain adaptation (DA) from Wikipedia to biomedical text and none-of-the-above (NOTA) detection. We evaluate state-of-the-art models and propose BERT-PAIR, a matching-based approach that significantly outperforms traditional prototypical networks in these realistic settings.

Introduction

The goal of few-shot learning is to recognize new patterns from just a handful of examples. In Relation Classification (RC)—identifying how two entities in a sentence are related (e.g., "born in" or "may treat")—the original FewRel dataset became the gold standard. However, models quickly saturated its performance, leading to a "human-level" illusion that ignores the noise and domain diversity of real-world data.

FewRel 2.0 raises the bar by introducing two major challenges. First, the Domain Gap requires models trained on general Wikipedia text to generalize to specialized biomedical literature (PubMed). Second, the "Closed-World" Assumption is removed via the NOTA challenge, where models must identify queries that do not belong to any of the provided support classes.

Illustration of the FewRel 2.0 task setup, showing the transition from Wikipedia training to biomedical testing and the inclusion of NOTA queries.

Methodology

To address these new hurdles, the researchers explored two primary strategies:

  • Adversarial Training for Domain Adaptation: This involves a "game" between an encoder that extracts features and a discriminator that tries to guess the source domain (Wikipedia vs. Biomedical). The encoder is trained to "fool" the discriminator, forcing it to learn domain-invariant features.
  • BERT-PAIR for NOTA Detection: Instead of traditional classification, BERT-PAIR treats RC as a matching problem. It pairs a query sentence with every provided support sentence and uses BERT to score the likelihood that they express the same relation. If scores across all known relations are low, the model labels the query as NOTA.

Datasets

FewRel 2.0 builds upon the original Wikipedia-based FewRel 1.0. While the training set remains the same, a new Biomedical Test Set was created by aligning PubMed literature with the UMLS knowledge base, resulting in 25 relations with 100 instances each. Additionally, the SemEval-2010 Task 8 dataset was utilized as a validation set to help bridge the gap between the general and specialized domains.

Key Results and Insights

The "Human-Level" Illusion

Models that previously "surpassed humans" on FewRel 1.0 saw their accuracy plummet by 30-50% when tested on the biomedical domain. This suggests that current models are often overfitting to specific linguistic styles rather than truly learning the abstract concept of relations. As shown in the performance comparison, the drop is consistent across both 1-shot and 5-shot settings.

Comparison of model performance on the original FewRel 1.0 vs. the new DA and NOTA tasks in FewRel 2.0, illustrating the significant performance gap.

Matching vs. Classifying

The BERT-PAIR approach demonstrated that for few-shot tasks, comparing examples (matching) is more robust than building a representative "prototype" for a class. On the biomedical test set, standard CNN-based Prototypical Networks achieved only ~35% accuracy, while BERT-PAIR reached 56.25% (1-shot) and 67.44% (5-shot). Furthermore, BERT-PAIR proved remarkably stable as the percentage of NOTA queries increased, whereas other models' accuracy degraded rapidly.

The Impact of In-Domain Data

The researchers found that adding just 1,000 training instances from the target biomedical domain boosted BERT-PAIR's performance by over 13 points. This highlights a massive "growth boundary" for future research in semi-supervised and unsupervised domain adaptation.

Conclusion and Future Work

FewRel 2.0 exposes the fragility of current few-shot RC models when faced with domain shifts and open-world queries. While BERT-PAIR and adversarial training offer significant improvements, a substantial performance gap remains compared to the original domain. Future work is needed to develop better unsupervised domain adaptation techniques and to improve NOTA detection by better characterizing the feature space of irrelevant relations.

Summary updated

Abstract

We present FewRel 2.0, a more challenging task to investigate two aspects of few-shot relation classification models: (1) Can they adapt to a new domain with only a handful of instances? (2) Can they detect none-of-the-above (NOTA) relations? To construct FewRel 2.0, we build upon the FewRel dataset (Han et al., 2018) by adding a new test set in a quite different domain, and a NOTA relation choice. With the new dataset and extensive experimental analysis, we found (1) that the state-of-the-art few-shot relation classification models struggle on these two aspects, and (2) that the commonly-used techniques for domain adaptation and NOTA detection still cannot handle the two challenges well. Our research calls for more attention and further efforts to these two real-world issues. All details and resources about the dataset and baselines are released at https: //github.com/thunlp/fewrel.
BibTeX
@inproceedings{gao2019fewrel,
  author = {Tianyu Gao and Xu Han and Hao Zhu and Zhiyuan Liu and Peng Li and Maosong Sun and Jie Zhou},
  title = {{FewRel 2.0: Towards More Challenging Few-Shot Relation Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1649},
}
Figure