Function-words Adaptively Enhanced Attention Networks for Few-Shot Inverse Relation Classification

Chunliu Dou, Shaojuan Wu, Xiaowang Zhang, Zhiyong Feng, Kewen Wang

Abstract

Relation classification (RC) identifies semantic relations between two entities within a given text. While current models perform well with abundant data, their performance degrades significantly in few-shot learning scenarios, especially when handling inverse relations. We propose the Function-words Adaptively Enhanced Attention (FAEA) framework, which employs a hybrid attention model to focus on class-related function words. To address the intra-class redundancy introduced by function words, we incorporate an adaptive message-passing mechanism to capture and transfer inter-class differences. Our results show that FAEA substantially improves performance on FewRel benchmarks, notably increasing inverse relation accuracy by 14.33% in 1-shot settings.

Introduction

Few-Shot Relation Classification (FSRC) aims to recognize novel relations using only a handful of labeled examples. A critical yet overlooked challenge in FSRC is the presence of inverse relations. In datasets like FewRel 1.0, over 21% of relations have an inverse counterpart (e.g., "A is part of B" vs. "B has part A").

Standard models typically focus on "content words" (nouns and verbs), which are often identical in inverse pairs. The semantic distinction in these cases is carried by "function words" like prepositions and auxiliary verbs. Current FSRC models often treat these as noise, leading to significant confusion between similar classes. The authors identify that while baseline models perform well on random tasks, their accuracy drops by nearly 27 points when forced to handle inverse relations in a 1-shot setting.

The FAEA Framework

The FAEA framework addresses the "inverse gap" through three primary components designed to elevate the importance of syntactic structure.

1. Instance Encoder

The model utilizes a standard BERT-based backbone to generate initial contextualized embeddings for the input sentence and the target entities. This provides a rich semantic foundation before the specialized attention layers are applied.

2. Function-words Enhanced Attention

This core component moves beyond simple keyword matching by implementing a hybrid attention strategy. Instead of a one-size-fits-all approach, FAEA uses:

  • Class-general attention: This learns which function words (like "of", "in", "by") are globally significant across various relation types in the training set.
  • Class-specific attention: This focuses on function words that are semantically or structurally related to the specific keywords of a relation. For example, it might highlight "of" specifically when the relation involves containment or part-whole structures.
The FAEA framework uses a hybrid attention mechanism to highlight critical function words (right) and a global attention heatmap (left) to distinguish between similar relation classes.

3. Adaptive Message Passing (AMP)

While function words provide discriminative signals, they also introduce intra-class variance (redundancy) because the same function words appear in many different contexts. To counter this, FAEA treats instances as nodes in a graph. The AMP mechanism passes messages between nodes to reinforce similarities within a class and exaggerate differences between different classes. Mathematically, this acts as a filter to reduce the noise inherent in high-frequency function words while keeping the "signal" that defines the relation direction.

Experimental Results

The model was evaluated on FewRel 1.0, FewRel 2.0, and a specialized subset called FSIRC (Few-Shot Inverse Relation Classification), which specifically samples tasks containing inverse relations.

Performance on Standard Benchmarks

On FewRel 1.0 (5-way 1-shot), FAEA improved accuracy by 4.80% over strong baselines. When combined with pre-training (CP), it reached a state-of-the-art 94.11%. On FewRel 2.0, which tests domain adaptation (e.g., moving from general text to medical or chemical domains), the model showed superior robustness, outperforming BERT-Pair by over 6% in 1-shot tasks.

Inverse Relation Mastery

The most striking results came from the FSIRC task. FAEA outperformed the next best model (ConceptFERE) by 14.33% in the 1-shot setting. This confirms the hypothesis that function words are the primary syntactic markers that define the direction of a relationship in low-data regimes.

Conclusion

FAEA demonstrates that function words, often discarded as stop words, are vital structural signals for few-shot learning. By combining hybrid attention with adaptive message passing, the framework effectively balances the need for discriminative syntactic features with the requirement for low intra-class variance. These findings suggest that future FSRC models should move away from purely content-word-centric approaches and embrace the structural nuances of language.

Summary updated

Abstract

The relation classification is to identify semantic relations between two entities in a given text. While existing models perform well for classifying inverse relations with large datasets, their performance is significantly reduced for few-shot learning. In this paper, we propose a function words adaptively enhanced attention framework (FAEA) for few-shot inverse relation classification, in which a hybrid attention model is designed to attend class-related function words based on meta-learning. As the involvement of function words brings in significant intra-class redundancy, an adaptive message passing mechanism is introduced to capture and transfer inter-class differences.We mathematically analyze the negative impact of function words from dot-product measurement, which explains why the message passing mechanism effectively reduces the impact. Our experimental results show that FAEA outperforms strong baselines, especially the inverse relation accuracy is improved by 14.33% under 1-shot setting in FewRel1.0.
BibTeX
@inproceedings{dou2022function,
  author = {Chunliu Dou and Shaojuan Wu and Xiaowang Zhang and Zhiyong Feng and Kewen Wang},
  title = {{Function-words Adaptively Enhanced Attention Networks for Few-Shot Inverse Relation Classification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=E4J0eN55DdY},
}
Figure