Abstract
Continual relation extraction (CRE) aims to continuously train a model on data with new relations while avoiding the catastrophic forgetting of previously learned ones. While memory-based methods—which store and replay typical samples of old relations—have shown promise, they often overfit to limited memory samples and struggle with imbalanced datasets. To address these challenges, we propose Consistent Representation Learning (CRL). This framework maintains the stability of relation embeddings by integrating supervised contrastive learning and knowledge distillation during memory replay. Specifically, we employ a memory-bank-based contrastive learning approach to learn distinct relation representations, followed by contrastive replay and memory knowledge distillation to preserve the semantic structure of historical tasks. Extensive experiments on the FewRel and TACRED datasets demonstrate that CRL significantly outperforms state-of-the-art baselines and exhibits strong robustness on imbalanced data.
Introduction
In Natural Language Processing (NLP), Relation Extraction (RE) is the task of identifying semantic connections between entities in text (e.g., identifying the founder relation in "Steve Jobs founded Apple"). While traditional models are trained on static datasets, real-world applications require models to learn new relations incrementally. This paradigm, known as Continual Relation Extraction (CRE), is hindered by catastrophic forgetting, where the optimization for new tasks erases the weights and knowledge associated with previous ones.
Existing memory-based strategies attempt to mitigate this by replaying a small subset of historical data. However, these methods often suffer from a "warping" of the embedding space—the mathematical map where relations are represented. As the model learns new tasks, the clusters for old relations shift and blur, leading to classification errors. CRL is designed to ensure "representation consistency," keeping the model's internal map stable even as new information is added.
Methodology
The CRL framework consists of four primary components designed to maintain a stable and discriminative embedding space:
1. Initial Training with Supervised Contrastive Learning (SCL)
When a new task is introduced, the model utilizes SCL to organize the embedding space. Unlike standard cross-entropy loss, SCL explicitly forces instances of the same relation to cluster tightly together while pushing instances of different relations apart. This creates a highly structured foundation for the new relations before they are integrated into the long-term memory.
2. Prototypical Sample Selection
To maximize the utility of the limited memory bank, CRL avoids random sampling. Instead, it employs k-means clustering to identify the most representative instances of a relation. These "prototypes" are stored in the memory bank, ensuring that the replay phase uses the most informative data points to represent the historical distribution.
3. Consistent Representation Learning
To prevent the embedding space from shifting during the integration of new tasks, CRL performs two simultaneous operations during replay:
- Contrastive Replay: The model re-runs contrastive learning across the entire memory bank (containing both old and new relations). This ensures that the boundaries between all known relations remain distinct.
- Memory Knowledge Distillation (KD): Using a teacher-student architecture, the model ensures that the relative distances between different relation prototypes remain consistent with previous states. By distilling this structural knowledge, the model preserves the semantic "map" it built during earlier tasks.
4. Nearest Class Mean (NCM) Classifier
Traditional linear classifiers are often biased toward the most recently learned tasks. To counter this, CRL utilizes an NCM classifier. During inference, a new sentence is classified by calculating its embedding and assigning it to the relation whose prototype (mean vector) is closest in the embedding space.
Experimental Setup
We evaluated CRL on two major benchmarks:
- FewRel: A balanced dataset with 80 relations and 700 instances per relation, testing the model's ability to handle a high volume of distinct categories.
- TACRED: A large-scale, highly imbalanced dataset with 42 relations. This serves as a rigorous test for real-world applicability, where some relations are significantly more frequent than others.
Results and Discussion
CRL established new state-of-the-art (SOTA) benchmarks across both datasets:
- Performance on FewRel: After 10 incremental tasks, CRL achieved an accuracy of 83.1%, surpassing the previous best model, RP-CRE (82.6%).
- Robustness on TACRED: The advantage of CRL was most evident on the imbalanced TACRED dataset, where it reached 78.0% accuracy—a 2.3% improvement over the previous SOTA (75.7%). This suggests that CRL's focus on representation stability is particularly effective when data distributions are skewed.
- Embedding Stability: t-SNE visualizations of the embedding space confirmed that while baseline models' clusters became scattered and overlapping as more tasks were added, CRL’s clusters remained tight and well-separated throughout the continual learning process.
Conclusion
Consistent Representation Learning (CRL) effectively addresses the stability-plasticity dilemma in Continual Relation Extraction. By combining supervised contrastive learning with memory knowledge distillation, the framework prevents the degradation of the embedding space. Future research will explore the application of CRL in cross-domain scenarios—such as transitioning from general news to specialized medical text—and investigate dynamic memory allocation strategies to further optimize performance.
Summary updated