Continual Few-shot Relation Extraction via Adaptive Gradient Correction and Knowledge Decomposition

Jianpeng Hu, Chengxiang Tan, Jiacheng Xu, Xiangyun KongACL 2024 · DOI

Introduction

Relation Extraction (RE) is a fundamental task in natural language processing, aimed at identifying semantic relationships between entities in text. In real-world scenarios, models often need to learn new relations incrementally with limited data—a setting known as Continual Few-shot Relation Extraction (CFRE). However, this process is plagued by catastrophic forgetting, where learning new tasks degrades performance on previously learned ones.

The authors of this paper identify embedding instability as a primary driver of forgetting. In few-shot learning, the "prototypes" (class centroids) and individual sample embeddings often drift inconsistently during training. This lack of "following degree" increases the Relative Offset Distance (ROD), causing the model to lose established decision boundaries.

Illustration of embedding instability where samples and prototypes drift apart, increasing Relative Offset Distance (ROD).

The AGCKD Framework

To address these challenges, the researchers propose AGCKD (Adaptive Gradient Correction and Knowledge Decomposition), a framework designed to stabilize embedding spaces and preserve knowledge through two primary mechanisms.

1. Adaptive Gradient Correction (AGC)

AGC prevents new updates from interfering with previously acquired knowledge. The model calculates a correction matrix based on "in-memory" samples from prior tasks. This matrix adaptively constrains gradients for the current task to be orthogonal to the embedding space of previous tasks. By ensuring updates to the Transformer’s Feed-Forward Networks (FFN) and Query matrices do not overlap with existing representations, the model effectively shields old knowledge.

2. Knowledge Decomposition (KD)

The architecture decouples general linguistic features from task-specific relation logic:

  • General Knowledge Encoder: Utilizes a Pre-trained Language Model (BERT) to capture broad language patterns.
  • Task-related Knowledge Encoder: Stores specific relation logic as "proto-embeddings."

This separation allows the model to refine task-specific boundaries without distorting the underlying general features. Training is further refined through a three-stage asynchronous optimization process that stabilizes class centers before fine-tuning proto-embeddings as learnable parameters.

Experimental Results

The AGCKD method was evaluated on two major benchmarks: FewRel (80 relations) and TACRED (41 relations).

Performance comparison of AGCKD against baselines on FewRel and TACRED datasets.

As shown in the results, AGCKD consistently outperforms state-of-the-art (SOTA) models like ConPL and ERDA across various settings (10-way 2-shot, 10-way 10-shot, and 5-way 10-shot). On the FewRel 10-way 5-shot benchmark, AGCKD achieved a final average accuracy of 88.68%, compared to ConPL's 85.83%. Notably, the forgetting rate for AGCKD was significantly lower (3.24%) than that of ERDA (12.78%), demonstrating the effectiveness of the gradient correction mechanism in maintaining performance on earlier tasks.

Conclusion

AGCKD introduces a robust approach to CFRE by focusing on embedding stability. Through adaptive gradient orthogonality and knowledge decomposition, the framework successfully mitigates catastrophic forgetting while maintaining high accuracy on new tasks. Future work will explore extending these gradient correction algorithms to non-Transformer architectures and incorporating inter-class distances to further optimize the embedding space.

Summary updated

BibTeX
@inproceedings{hu2024continual,
  author = {Jianpeng Hu and Chengxiang Tan and Jiacheng Xu and Xiangyun Kong},
  title = {{Continual Few-shot Relation Extraction via Adaptive Gradient Correction and Knowledge Decomposition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.702},
}
Figure