fmLRE: A Low-Resource Relation Extraction Model Based on Feature Mapping Similarity Calculation

Peng Wang, Tong Shao, Ke Ji, Guozheng Li, Wenjun KeAAAI 2023 · OpenReview

Abstract

Low-resource relation extraction (LRE) aims to extract relations from limited labeled corpora. Existing work takes advantages of self-training or distant supervision to expand the limited labeled data in the data-driven approaches, while the selection bias of pseudo labels may cause the error accumulation in subsequent relation classification. To address this issue, this paper proposes fmLRE, an iterative feedback method based on feature mapping similarity calculation to improve the accuracy of pseudo labels. First, it calculates the similarities between pseudo-label and real-label data of the same category in a feature mapping space based on semantic features of labeled dataset after feature projection. Then, it fine-tunes the initial model according to the iterative process of reinforcement learning. Finally, the similarity is used as a threshold for screening high-precision pseudo-labels and the basis for setting different rewards, which also acts as a penalty term for the loss function of the relation classifier. Experimental results demonstrate that fmLRE achieves state-of-the-art performance compared with strong baselines on two public datasets.

Introduction

Relation Extraction (RE) is the task of identifying semantic relationships between entities in text, such as recognizing that "Steve Jobs" and "Apple" share a "founder of" relationship. While modern deep learning models excel at this when trained on massive datasets, they struggle in low-resource scenarios where labeled data is scarce.

A common solution is "self-training," where a model labels unlabeled data (creating "pseudo-labels") to expand its own training set. However, this often leads to selection bias: if the initial model makes a mistake, it trains on its own errors, leading to a cycle of error accumulation. The fmLRE framework was developed to break this cycle by creating a more robust way to validate and utilize these pseudo-labels.

Methodology

The fmLRE framework is an iterative feedback method that uses two primary "engines" to improve label quality:

The fmLRE architecture illustrating the flow from feature extraction to the reinforcement learning feedback loop.

Feature Mapping Similarity (Metric Learning)

Instead of just trusting the model's classification output, fmLRE projects sentences into a specialized "feature mapping space." It calculates the similarity between a new pseudo-labeled sample (the "query") and the average feature of known samples in that category (the "support set"). If the similarity is high, the label is trusted. This category-aware approach is more sensitive to relational nuances than global similarity measures.

Reinforcement Learning (RL) Feedback

The similarity score is used as a reward signal in an RL framework. High similarity provides positive feedback, while low similarity acts as a penalty term in the loss function. This forces the model to align its internal representations more closely with real-world label categories.

Noise Filtering and Model Backbone

The model includes a "strike system" for unlabeled data. If a sample repeatedly fails to meet the similarity threshold over several iterations, it is flagged as noise and removed. Additionally, replacing a standard BERT backbone with Matching the Blanks (MTB)—a model pre-trained specifically for relational goals—significantly raises the performance ceiling.

Experimental Results

The model was evaluated on the SemEval 2010 Task 8 and TACRED datasets. fmLRE consistently outperformed strong baselines like GradLRE and MetaSRE across different ratios of labeled data.

Performance comparison on SemEval and TACRED datasets, showing fmLRE's superiority in low-resource settings.

On SemEval, with only 5% labeled data, fmLRE achieved an 82.11% F1 score, outperforming the next best baseline by 2.5%. On the challenging TACRED dataset, in a very sparse 3% labeled data scenario, fmLRE reached 50.83% F1, a significant jump over GradLRE’s 47.37%.

Discussion and Conclusion

The core innovation of fmLRE is using similarity in a mapped space for three distinct purposes: as a gatekeeper (filtering labels), a reward (for RL), and a penalty (for the classifier loss). While highly effective, the authors acknowledge limitations such as the "zero-shot gap"—the inability to identify relations never seen during training—and the current need for manual tuning of the similarity threshold. Future work will explore adaptive learning to automate threshold selection and improve generalization for entirely novel relations.

Summary updated

Abstract

Low-resource relation extraction (LRE) aims to extract relations from limited labeled corpora. Existing work takes advantages of self-training or distant supervision to expand the limited labeled data in the data-driven approaches, while the selection bias of pseudo labels may cause the error accumulation in subsequent relation classification. To address this issue, this paper proposes fmLRE, an iterative feedback method based on feature mapping similarity calculation to improve the accuracy of pseudo labels. First, it calculates the similarities between pseudo-label and real-label data of the same category in a feature mapping space based on semantic features of labeled dataset after feature projection. Then, it fine-tunes initial model according to the iterative process of reinforcement learning. Finally, the similarity is used as a threshold for screening high-precision pseudo-labels and the basis for setting different rewards, which also acts as a penalty term for the loss function of relation classifier. Experimental results demonstrate that fmLRE achieves the state-of-the-art performance compared with strong baselines on two public datasets.
BibTeX
@inproceedings{wang2023fmlre,
  author = {Peng Wang and Tong Shao and Ke Ji and Guozheng Li and Wenjun Ke},
  title = {{fmLRE: A Low-Resource Relation Extraction Model Based on Feature Mapping Similarity Calculation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=cQHWDLf7pi},
}
Figure