Abstract
Few-shot relation classification (RC) models have reached a performance plateau on existing benchmarks, often surpassing human-level accuracy on "clean" datasets like FewRel 1.0. However, these benchmarks frequently fail to reflect real-world complexities such as domain shifts and irrelevant queries. This paper introduces FewRel 2.0, a more challenging dataset that incorporates two grueling real-world hurdles: domain adaptation (DA) from Wikipedia to biomedical text and none-of-the-above (NOTA) detection. We evaluate state-of-the-art models and propose BERT-PAIR, a matching-based approach that significantly outperforms traditional prototypical networks in these realistic settings.
Introduction
The goal of few-shot learning is to recognize new patterns from just a handful of examples. In Relation Classification (RC)—identifying how two entities in a sentence are related (e.g., "born in" or "may treat")—the original FewRel dataset became the gold standard. However, models quickly saturated its performance, leading to a "human-level" illusion that ignores the noise and domain diversity of real-world data.
FewRel 2.0 raises the bar by introducing two major challenges. First, the Domain Gap requires models trained on general Wikipedia text to generalize to specialized biomedical literature (PubMed). Second, the "Closed-World" Assumption is removed via the NOTA challenge, where models must identify queries that do not belong to any of the provided support classes.
Methodology
To address these new hurdles, the researchers explored two primary strategies:
- Adversarial Training for Domain Adaptation: This involves a "game" between an encoder that extracts features and a discriminator that tries to guess the source domain (Wikipedia vs. Biomedical). The encoder is trained to "fool" the discriminator, forcing it to learn domain-invariant features.
- BERT-PAIR for NOTA Detection: Instead of traditional classification, BERT-PAIR treats RC as a matching problem. It pairs a query sentence with every provided support sentence and uses BERT to score the likelihood that they express the same relation. If scores across all known relations are low, the model labels the query as NOTA.
Datasets
FewRel 2.0 builds upon the original Wikipedia-based FewRel 1.0. While the training set remains the same, a new Biomedical Test Set was created by aligning PubMed literature with the UMLS knowledge base, resulting in 25 relations with 100 instances each. Additionally, the SemEval-2010 Task 8 dataset was utilized as a validation set to help bridge the gap between the general and specialized domains.
Key Results and Insights
The "Human-Level" Illusion
Models that previously "surpassed humans" on FewRel 1.0 saw their accuracy plummet by 30-50% when tested on the biomedical domain. This suggests that current models are often overfitting to specific linguistic styles rather than truly learning the abstract concept of relations. As shown in the performance comparison, the drop is consistent across both 1-shot and 5-shot settings.
Matching vs. Classifying
The BERT-PAIR approach demonstrated that for few-shot tasks, comparing examples (matching) is more robust than building a representative "prototype" for a class. On the biomedical test set, standard CNN-based Prototypical Networks achieved only ~35% accuracy, while BERT-PAIR reached 56.25% (1-shot) and 67.44% (5-shot). Furthermore, BERT-PAIR proved remarkably stable as the percentage of NOTA queries increased, whereas other models' accuracy degraded rapidly.
The Impact of In-Domain Data
The researchers found that adding just 1,000 training instances from the target biomedical domain boosted BERT-PAIR's performance by over 13 points. This highlights a massive "growth boundary" for future research in semi-supervised and unsupervised domain adaptation.
Conclusion and Future Work
FewRel 2.0 exposes the fragility of current few-shot RC models when faced with domain shifts and open-world queries. While BERT-PAIR and adversarial training offer significant improvements, a substantial performance gap remains compared to the original domain. Future work is needed to develop better unsupervised domain adaptation techniques and to improve NOTA detection by better characterizing the feature space of irrelevant relations.
Summary updated