Abstract: To address the phenomenon of similar classes, existing methods in few-shot continual relation extraction (FCRE) face two main challenges: non-representative prototypes and representation bias, especially when the number of available samples is limited. In our work, we propose Minion to address these challenges. Firstly, we leverage the General Orthogonal Frame (GOF) structure, based on the concept of Neural Collapse, to create robust class prototypes with clear separation, even between analogous classes. Secondly, we utilize label description representations as global class representatives within the fast-slow contrastive learning paradigm. These representations consistently encapsulate the essential attributes of each relation, acting as global information that helps mitigate overfitting and reduces representation bias caused by the limited local few-shot examples within a class. Extensive experiments on well-known FCRE benchmarks show that our method outperforms state-of-the-art approaches, demonstrating its effectiveness for advancing RE systems.
Introduction
In real-world NLP systems, models must continuously learn new relationships between entities without forgetting previously learned ones—a challenge known as Continual Relation Extraction (CRE). This is notoriously difficult in few-shot scenarios (FCRE), where the model only gets a handful of examples for each new relation. Existing FCRE methods struggle with two main issues: non-representative prototypes, where averaging a few embeddings fails to distinguish similar relations, and representation bias, where models discard nuanced features crucial for future tasks to solve the current task efficiently.
The Minion Framework
To solve these challenges, we introduce Minion, a framework consisting of two core innovations: ProtoGOF and FCLD.
ProtoGOF: Guiding Prototypes via General Orthogonal Frames
While recent deep learning theory suggests that class features often collapse into an Equiangular Tight Frame (ETF) at the end of training, ETF is too rigid for continual learning as it requires knowing the total number of classes in advance. Minion instead utilizes a General Orthogonal Frame (GOF). GOF enforces orthogonal separation between class prototypes while remaining flexible enough to dynamically add new classes and handle imbalanced data as new tasks arrive.
FCLD: Fast-Slow Contrastive Learning with Label Descriptions
To address representation bias, Minion employs a dual-encoder architecture consisting of a "fast" updating encoder and a "slow" momentum-updated encoder. Instead of relying solely on limited input sentences, Minion feeds textual descriptions of relation labels into the slow encoder. These descriptions act as stable, global semantic anchors, preventing the model from overfitting to the quirks of limited few-shot training examples.
Experimental Results
We evaluated Minion on two standard FCRE benchmarks: FewRel (8 tasks, 10-way 5-shot) and TACRED (8 tasks, 5-way 5-shot). Our experiments included various backbones, ranging from BERT to modern Large Language Model Embeddings (LLMEs) like LLaMA-2, LLaMA-3, and Mistral.
Key Findings
- State-of-the-Art Performance: Using a standard BERT backbone, Minion achieved 69.61% accuracy on FewRel and 61.11% on TACRED, outperforming previous SOTA models.
- Scalability to LLMs: The architecture scales effectively with LLMs. With LLaMA-3, Minion reached 81.01% on FewRel, a significant +2.19% improvement over the best baseline.
- Ablation Studies: Removing either ProtoGOF or FCLD resulted in a 1-2% performance drop. Qualitative analysis showed that ProtoGOF drastically reduces misclassifications between highly similar relations (e.g., "Publisher" vs. "Screenwriter").
Conclusion
Minion effectively mitigates non-representative prototypes and representation bias in FCRE by combining dynamic orthogonal prototypes with global label description anchors. While the current framework assumes predefined entities, future work will explore end-to-end extraction and the integration of data augmentation techniques to further enhance performance.