Towards Realistic Few-Shot Relation Extraction

Sam Brody, Sichao Wu, Adrian BentonEMNLP 2021 · OpenReview

Towards Realistic Few-Shot Relation Extraction

Abstract: Sam Brody, Sichao Wu, Adrian Benton. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

2021.

Few-shot relation extraction (RE) models have recently shown performance levels approaching human benchmarks. However, this paper demonstrates that these high scores often result from models exploiting simple entity type signatures rather than understanding linguistic context. By introducing a more challenging evaluation framework and a targeted data augmentation strategy, the researchers show how to force models toward a more robust understanding of relational context.

The Entity Type Shortcut in Few-Shot RE

The standard benchmark for few-shot RE, FewRel, often sees models achieving over 90% accuracy. The authors argue this is misleading. In many benchmark scenarios, a relation can be uniquely identified simply by the types of entities involved (e.g., [Person] and [City]). If only one such relation exists in the test set, the model can "cheat" by matching entity types without reading the sentence.

In a realistic setting, a model must distinguish between relations with identical entity signatures, such as born_in, lived_in, and died_in. Standard N-way classification tasks in benchmarks like FewRel do not sufficiently penalize models for failing these nuanced distinctions.

Methodology: Beyond Simple Classification

To expose these weaknesses, the researchers employed Prototypical Networks using state-of-the-art transformers (BERT, RoBERTa, SpanBERT, and LUKE). Instead of standard classification, they implemented a Ranking Evaluation using Precision at 50 (P@50). This metric measures how many of the top 50 most similar sentences retrieved for a relation prototype actually belong to that relation.

Table 1: Accuracy of various encoders on the FewRel benchmark. Note the high performance of RoBERTa-large.

Furthermore, they proposed a Data Augmentation strategy. By enriching training data with "confusable" relations—those sharing entity type signatures—they aimed to force models to attend to the surrounding context rather than relying on entity types.

Key Findings and Results

The discrepancy between benchmark performance and realistic ranking is stark. While RoBERTa-large achieves a 96.23% accuracy on FewRel (see Table 1), its precision on specific "hard" relations is significantly lower. For example, the part_of relation yields only a 36.2% P@50.

Table 2: Baseline P@50 scores and top confounders for various relation types. Many relations are frequently confused with others sharing the same entity types.

The "Mother/Child/Spouse" problem exemplifies this: models struggle to distinguish these relations because the entity types ([Person], [Person]) are identical. On the spouse relation, RoBERTa’s precision was found to be as low as 43.2% in baseline tests due to confusion with mother or child examples.

The proposed augmentation using TACRED "person" relations significantly improved performance on unrelated "organization" relations. For instance, precision on the hq_city (headquartered in city) relation jumped from 59.6% to 84.4%, representing a 24.8% absolute increase. This suggests that learning to distinguish difficult relations in one domain transfers a general contextual-awareness strategy to others.

Conclusion

This research highlights a critical bias in current few-shot RE models toward entity type signatures. While data augmentation provides a significant performance boost, the authors suggest that future work should focus on architectural changes or smarter training samplers that intentionally present clusters of confusable relations to the model.

Summary updated

BibTeX
@inproceedings{brody2021towards,
  author = {Sam Brody and Sichao Wu and Adrian Benton},
  title = {{Towards Realistic Few-Shot Relation Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=BtUWpn3KMzc},
}
Figure