Consistent Representation Learning for Continual Relation Extraction

Kang Zhao, Hua Xu, Jiangong Yang, Kai Gao

Abstract

Continual Relation Extraction (CRE) aims to learn a sequence of relation extraction tasks without suffering from catastrophic forgetting. Traditional methods often struggle with maintaining a stable representation space, especially when faced with imbalanced data. We present Consistent Representation Learning (CRL), a framework that combines supervised contrastive learning with knowledge distillation to maintain a stable and organized embedding space. By anchoring the relative distances between relation prototypes, CRL ensures that the model's internal "map" of relations remains consistent over time. Experimental results on the FewRel and TACRED benchmarks show that CRL achieves state-of-the-art performance, demonstrating significant robustness on imbalanced real-world datasets.

Introduction

Relation Extraction (RE) is a fundamental task in Natural Language Processing (NLP) that involves identifying the semantic connection between two entities in a sentence. While standard models excel on fixed datasets, real-world applications require models to learn new relations incrementally. This process, known as Continual Relation Extraction (CRE), is often hindered by catastrophic forgetting, where the acquisition of new knowledge leads to the sudden loss of previously learned information.

Existing memory-based approaches attempt to mitigate this by storing a small subset of old examples for replay. However, these methods frequently overfit to the limited memory samples or fail to maintain a coherent representation space when data distributions are imbalanced. CRL addresses these challenges by focusing on the stability and geometry of the embedding space.

The CRL Framework

The Consistent Representation Learning (CRL) framework operates through a three-phase process for each incoming task:

  1. Supervised Contrastive Learning (SCL): Instead of relying solely on classification boundaries, CRL uses a contrastive loss to pull similar relation embeddings together and push dissimilar ones apart. This creates a robust, high-dimensional map of relations.
  2. Representative Sample Selection: Following the training of a new task, the model employs k-means clustering to identify the most representative instances of each relation to be stored in a compact memory bank.
  3. Consistency Maintenance: To integrate new tasks without disrupting old ones, CRL performs two simultaneous operations: Contrastive Replay, which uses memory bank samples to keep existing clusters tight, and Knowledge Distillation (KD). The KD component acts as a structural anchor, ensuring that the relative distances between different relation prototypes remain stable even as new relations are introduced.
The Consistent Representation Learning (CRL) framework architecture, illustrating the integration of supervised contrastive loss and knowledge distillation.

For final prediction, CRL utilizes a Nearest Class Mean (NCM) classifier. By comparing new inputs to the established cluster centers (prototypes) rather than using a standard linear layer, the model avoids the bias typically associated with the most recently learned tasks.

Experimental Evaluation

CRL was evaluated on two primary benchmarks:

  • FewRel: A balanced dataset consisting of 80 relations with 700 instances each.
  • TACRED: A large-scale, highly imbalanced dataset with 42 relations, representing a more challenging real-world scenario where sample sizes vary significantly across classes.

Results and Discussion

CRL established new state-of-the-art (SOTA) benchmarks across both datasets. On the FewRel dataset, CRL achieved an accuracy of 84.4% after 10 tasks, surpassing the previous best model, RP-CRE. The performance gain was even more pronounced on the imbalanced TACRED dataset, where CRL reached 80.7% accuracy, a 3.4% improvement over the previous SOTA.

Performance comparison of CRL against baseline models on the FewRel and TACRED datasets across 10 tasks.

Key insights from the results include:

  • Geometric Stability: The success of CRL highlights that maintaining the geometry of the embedding space is as critical as remembering specific examples.
  • Contrastive Resilience: Contrastive learning proved more resilient than traditional cross-entropy loss in continual learning settings by focusing on the underlying data structure.
  • Memory Efficiency: CRL demonstrated high performance even with minimal memory (e.g., 5 samples per relation), showing less dependence on large memory buffers than competing models.

Conclusion

Consistent Representation Learning (CRL) effectively addresses the forgetting problem in CRE by ensuring the stability of the model's representation space. By combining contrastive learning with structural anchoring via distillation, CRL provides a robust solution for incremental learning, particularly in imbalanced environments. Future research will explore the application of CRL in cross-domain scenarios and its scalability over longer sequences of tasks.

Summary updated

Abstract

Continual relation extraction (CRE) aims to continuously train a model on data with new relations while avoiding forgetting old ones. Some previous work has proved that storing a few typical samples of old relations and replaying them when learning new relations can effectively avoid forgetting. However, these memory-based methods tend to overfit the memory samples and perform poorly on imbalanced datasets. To solve these challenges, a consistent representation learning method is proposed, which maintains the stability of the relation embedding by adopting contrastive learning and knowledge distillation when replaying memory. Specifically, supervised contrastive learning based on a memory bank is first used to train each new task so that the model can effectively learn the relation representation. Then, contrastive replay is conducted of the samples in memory and makes the model retain the knowledge of historical relations through memory knowledge distillation to prevent the catastrophic forgetting of the old task. The proposed method can better learn consistent representations to alleviate forgetting effectively. Extensive experiments on FewRel and TACRED datasets show that our method significantly outperforms state-of-the-art baselines and yield strong robustness on the imbalanced dataset.
BibTeX
@inproceedings{zhao2022consistent,
  author = {Kang Zhao and Hua Xu and Jiangong Yang and Kai Gao},
  title = {{Consistent Representation Learning for Continual Relation Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=17nnWhFTBo},
}
Figure