Adaptive Prototypical Networks With Label Words and Joint Representation Learning for Few-Shot Relation Classification

Yan Xiao, Yaochu Jin, Kuangrong Hao

TL;DR

This paper tackles the challenge of few-shot relation classification (FSRC) in natural language processing, where models must identify relationships between entities with very limited training examples for new relation types. They propose an "Adaptive Prototypical Networks with Label Words and Joint Representation Learning" (APN-LW-JRL) that enhances class prototypes by incorporating label words and uses a joint learning objective to improve representation quality, achieving state-of-the-art results.

What Problem Does This Paper Address?

Relation Classification (RC) is a crucial task in information extraction, aiming to identify the relationships between pairs of entities within text. While traditional supervised methods require vast amounts of labeled data, distant supervision, which leverages existing knowledge bases, introduces noise and struggles with the "long-tail" problem (many relation types with very few examples). This paper focuses on Few-Shot Relation Classification (FSRC), where models need to generalize to new, unseen relation categories with only a handful of examples. This is analogous to how humans can learn new concepts from minimal exposure, a capability that current deep learning models often lack, especially in the complex domain of natural language.

Methods

The core of the proposed method, APN-LW-JRL, builds upon Prototypical Networks, a popular metric-learning approach for few-shot learning. The authors introduce two key innovations:

  1. Adaptive Label Information Mixture Mechanism: Instead of solely relying on the average of support samples to form a class prototype, this mechanism adaptively incorporates the semantic information of label words (e.g., "country of citizenship," "developer"). This is achieved through a convex combination where the mixing coefficient is learned based on the semantic embedding of the label description. This aims to create more informative and interactive class prototypes.
  2. Joint Representation Learning (JRL): To improve the discriminative power of the learned embedding space, the authors propose a joint training objective. This objective combines a standard cross-entropy loss for classification with a representation learning loss inspired by triplet loss. The representation loss encourages samples from the same class to be closer and samples from different classes to be farther apart, specifically by using "hardest triplets" (farthest positive and nearest negative instances) to learn a more robust embedding.

The overall architecture involves an instance encoder (embedding + CNN) to process input sentences, which then feeds into the prototypical networks. The label words are also embedded and used in the adaptive mixture mechanism to refine the class prototypes.

Datasets

The primary dataset used for evaluation is FewRel. This dataset is designed for few-shot relation classification, generated through distant supervision and then filtered by crowdsourcing to reduce noise. It contains 70,000 instances across 100 relations, with each relation having approximately 700 instances. For their experiments, the authors use a subset of 80 relations, split into 48 for training, 12 for validation, and 20 for testing.

Key Insights and Ideas

  • Label words are crucial for prototypes: The paper highlights that simply averaging support samples for prototypes isn't always sufficient, especially when classes are similar. Incorporating the semantic meaning of label words directly into prototype formation significantly improves performance.
  • Label descriptions can be noisy: Counter-intuitively, the authors found that using label descriptions (longer text) was less effective than using shorter label words. They attribute this to potential ambiguity or metaphysical descriptions in the dataset, suggesting that concise, idiomatic label words are more beneficial for prototype representation.
  • Joint learning for better embeddings: Combining classification loss with a representation learning loss (inspired by triplet loss) leads to a more discriminative embedding space, which is critical for few-shot learning where distinguishing subtle differences is key.
  • Adaptive mechanisms are key to generalization: The proposed adaptive mixture and joint learning mechanisms allow the model to better leverage limited data, leading to improved generalization capabilities on unseen classes.

Key Results

The proposed APN-LW-JRL model consistently outperforms several state-of-the-art baselines on the FewRel dataset across various few-shot settings (5-way 1-shot, 5-way 5-shot, 10-way 1-shot, 10-way 5-shot). For instance, in the 5-way 5-shot setting, APN-LW-JRL achieved an accuracy of 87.27%, significantly higher than ProNet (86.62%) and Pro-HATT (86.01%). Ablation studies confirmed that both the adaptive label information mixture mechanism and the joint representation learning contributed positively to the performance gains. Visualizations also showed that APN-LW-JRL learns more discriminative sentence embeddings.

Limitations and Future Work

The authors acknowledge that the effectiveness of label words over label descriptions might be dataset-specific and influenced by the quality of descriptions. They also note that their model was tested only on the FewRel dataset.

Future work includes:

  • Extending the model to zero-shot learning.
  • Validating the model on additional text datasets.
  • Investigating the application of few-shot learning for joint relation extraction and named entity recognition (NER).
  • Exploring applications in medical and other real-world problems.

Summary updated

Abstract

Relation classification (RC) task is one of fundamental tasks of information extraction, aiming to detect the relation information between entity pairs in unstructured natural language text and generate structured data in the form of entity–relation triple. Although distant supervision methods can effectively alleviate the problem of lack of training data in supervised learning, they also introduce noise into the data and still cannot fundamentally solve the long-tail distribution problem of the training instances. In order to enable the neural network to learn new knowledge through few instances such as humans, this work focuses on few-shot relation classification (FSRC), where a classifier should generalize to new classes that have not been seen in the training set, given only a number of samples for each class. To make full use of the existing information and get a better feature representation for each instance, we propose to encode each class prototype in an adaptive way from two aspects. First, based on the prototypical networks, we propose an adaptive mixture mechanism to add label words to the representation of the class prototype, which, to the best of our knowledge, is the first attempt to integrate the label information into features of the support samples of each class so as to get more interactive class prototypes. Second, to more reasonably measure the distances between samples of each category, we introduce a loss function for joint representation learning (JRL) to encode each support instance in an adaptive manner. Extensive experiments have been conducted on FewRel under different few-shot (FS) settings, and the results show that the proposed adaptive prototypical networks with label words and JRL has not only achieved significant improvements in accuracy but also increased the generalization ability of FSRC.
BibTeX
@article{xiao2023adaptive,
  author = {Yan Xiao and Yaochu Jin and Kuangrong Hao},
  title = {{Adaptive Prototypical Networks With Label Words and Joint Representation Learning for Few-Shot Relation Classification}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2023},
  doi = {10.1109/tnnls.2021.3105377},
}
Figure