Introduction
Relation Extraction (RE) is a fundamental task in natural language processing, aimed at identifying semantic relationships between entities in text. In real-world scenarios, models often need to learn new relations incrementally with limited data—a setting known as Continual Few-shot Relation Extraction (CFRE). However, this process is plagued by catastrophic forgetting, where learning new tasks degrades performance on previously learned ones.
The authors of this paper identify embedding instability as a primary driver of forgetting. In few-shot learning, the "prototypes" (class centroids) and individual sample embeddings often drift inconsistently during training. This lack of "following degree" increases the Relative Offset Distance (ROD), causing the model to lose established decision boundaries.
The AGCKD Framework
To address these challenges, the researchers propose AGCKD (Adaptive Gradient Correction and Knowledge Decomposition), a framework designed to stabilize embedding spaces and preserve knowledge through two primary mechanisms.
1. Adaptive Gradient Correction (AGC)
AGC prevents new updates from interfering with previously acquired knowledge. The model calculates a correction matrix based on "in-memory" samples from prior tasks. This matrix adaptively constrains gradients for the current task to be orthogonal to the embedding space of previous tasks. By ensuring updates to the Transformer’s Feed-Forward Networks (FFN) and Query matrices do not overlap with existing representations, the model effectively shields old knowledge.
2. Knowledge Decomposition (KD)
The architecture decouples general linguistic features from task-specific relation logic:
- General Knowledge Encoder: Utilizes a Pre-trained Language Model (BERT) to capture broad language patterns.
- Task-related Knowledge Encoder: Stores specific relation logic as "proto-embeddings."
This separation allows the model to refine task-specific boundaries without distorting the underlying general features. Training is further refined through a three-stage asynchronous optimization process that stabilizes class centers before fine-tuning proto-embeddings as learnable parameters.
Experimental Results
The AGCKD method was evaluated on two major benchmarks: FewRel (80 relations) and TACRED (41 relations).
As shown in the results, AGCKD consistently outperforms state-of-the-art (SOTA) models like ConPL and ERDA across various settings (10-way 2-shot, 10-way 10-shot, and 5-way 10-shot). On the FewRel 10-way 5-shot benchmark, AGCKD achieved a final average accuracy of 88.68%, compared to ConPL's 85.83%. Notably, the forgetting rate for AGCKD was significantly lower (3.24%) than that of ERDA (12.78%), demonstrating the effectiveness of the gradient correction mechanism in maintaining performance on earlier tasks.
Conclusion
AGCKD introduces a robust approach to CFRE by focusing on embedding stability. Through adaptive gradient orthogonality and knowledge decomposition, the framework successfully mitigates catastrophic forgetting while maintaining high accuracy on new tasks. Future work will explore extending these gradient correction algorithms to non-Transformer architectures and incorporating inter-class distances to further optimize the embedding space.
Summary updated