Prototypical Replay with Old-class Focusing Knowledge Distillation for Incremental Named Entity Recognition

Zesheng Liu, Qiannan Zhu, Cuiping Li, Hong Chen

Abstract

Incremental Named Entity Recognition (INER) is a critical task for maintaining and updating NLP models as new entity types emerge. However, these models frequently suffer from catastrophic forgetting, where learning new classes degrades performance on previously learned ones. This paper introduces POF (Prototypical Replay with Old-class Focusing Knowledge Distillation), a framework designed to mitigate forgetting by replaying compact statistical prototypes and utilizing a targeted distillation loss. Evaluations on major benchmarks including Few-NERD, I2B2, and OntoNotes5 show that POF consistently outperforms state-of-the-art methods, achieving up to a 9.73% improvement in final F1-scores.

Introduction

Traditional Named Entity Recognition (NER) models are typically trained on a static set of labels. In dynamic real-world environments, new entity types appear frequently, necessitating incremental updates. The primary challenge in INER is catastrophic forgetting, often caused by "classification confusion" where old entities are misidentified as background noise or confused with new types. While replaying raw training data can help, it is often impractical due to storage constraints and privacy concerns. This research proposes a more efficient approach using statistical summaries and focused knowledge transfer.

Methodology

Overview of the POF framework, illustrating the interaction between Prototypical Replay and Old-class Focusing Knowledge Distillation.

The POF framework consists of three primary components designed to balance stability and plasticity:

1. Prototypical Replay (PR)

Instead of storing raw text samples, POF saves the "essence" of each entity type. For every learned class, the model stores the mean and standard deviation of its feature vectors (prototypes). During incremental training steps, the model generates synthetic features based on these Gaussian distributions. Replaying these prototypes allows the model to revisit old class boundaries frequently at a negligible storage cost.

2. Old-class Focusing Knowledge Distillation (OFKD)

Standard Knowledge Distillation (KD) can be overly restrictive, hindering the model's ability to learn new information. OFKD addresses this by applying a Mean Squared Error (MSE) constraint specifically to the regions of input text identified as old entity types. This ensures that the feature representations critical for old classes remain stable without sacrificing the flexibility needed to adapt to new data.

3. Pseudo-Labeling

To prevent the model from treating old entities in new datasets as the "O" (non-entity) tag, the previous model iteration is used to generate soft labels for old entities in the current training set. This maintains the continuity of old knowledge across updates.

Experimental Results

The researchers evaluated POF against several competitive baselines, including RDP and IS3, across various incremental settings (e.g., FG-8-PG-1, where 8 classes are learned initially followed by 1 class per step).

Performance comparison (Macro-F1) on the OntoNotes5 dataset under the FG-8-PG-1 incremental setting.

Key Findings:

  • Superior Performance: POF achieved a 9.73% improvement in final task F1-score on the Few-NERD dataset and showed significant gains of up to 7.20% on I2B2 and OntoNotes5.
  • Importance of Variance: Ablation studies revealed that replaying both the mean and standard deviation of features is crucial. The added "noise" from the standard deviation helps the model develop more robust discriminative boundaries.
  • Efficiency: By replaying prototypes rather than raw exemplars, POF maintains high performance while remaining significantly more storage-efficient than traditional replay methods.

Conclusion

POF provides a robust solution for Incremental NER by combining the efficiency of prototypical replay with the precision of focused knowledge distillation. By protecting old-class feature regions and re-simulating class distributions, the model effectively navigates the stability-plasticity trade-off. Future work may explore more complex distribution modeling, such as Gaussian Mixture Models, to better capture the diversity of complex entity types.

Summary updated

Abstract

Catastrophic forgetting is a key challenge in incremental named entity recognition (INER). Existing methods often address this issue through distillation-based approaches, which involve transferring previously learned knowledge from the old model to the new one. However, these methods may not fully equip the new model with an adequate understanding of the characteristics about old entity types, leading to confusion when classifying tokens associated with these entity types. To address this challenge, we propose a novel method called Prototypical Replay with Old-class Focusing Knowledge Distillation (POF) for INER. Our approach focuses on preserving the main characteristics of each previous entity type by storing compact prototypes and replaying them with appropriate frequency. This replay strategy makes the new model review the knowledge of old entity types while minimizing storage needs. Additionally, we introduce an old-class focusing knowledge distillation (OFKD) loss, which distills features only in old-class regions to maintain the quality of old-class prototypes and prevent ineffective prototypical replay while preserving sufficient plasticity for learning new entity types. We conducted experiments on three benchmark datasets (i.e., Few-NERD, I2B2 and OntoNotes5), and the results demonstrate that our method outperforms all previous state-of-the-art methods.
BibTeX
@inproceedings{liu2025prototypical,
  author = {Zesheng Liu and Qiannan Zhu and Cuiping Li and Hong Chen},
  title = {{Prototypical Replay with Old-class Focusing Knowledge Distillation for Incremental Named Entity Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34651},
}
Figure