A Neural Span-Based Continual Named Entity Recognition Model

Yunan Zhang, Qingcai ChenAAAI 2023 · DOI

Abstract

Continual Named Entity Recognition (CL-NER) requires models to learn new entity types sequentially without forgetting previous ones. Traditional sequence-labeling approaches suffer from "catastrophic forgetting" due to incoherent optimization, where the same token's label changes across tasks (e.g., from "Outside" to a specific entity type). We introduce SpanKL, a span-based model that reformulates NER as independent binary classification for each span. This architecture decouples entity types, allowing for more stable incremental learning. Our results on OntoNotes and Few-NERD benchmarks demonstrate that SpanKL significantly outperforms state-of-the-art sequence-labeling baselines in both performance and stability.

Introduction: The Problem of Incoherent Optimization

In real-world scenarios, information extraction systems must adapt to new domains by recognizing additional entity types. Retraining from scratch is often computationally prohibitive or impossible due to data privacy. This necessitates Continual Named Entity Recognition (CL-NER).

The primary obstacle in CL-NER is catastrophic forgetting. In traditional sequence labeling (e.g., IOB tagging), every token must be assigned a label. When a model learns a new entity type, tokens that were previously labeled as O (Outside) might now be labeled as a new entity (e.g., B-PER). This "label flipping" creates a conflict: the model is penalized for predicting the old label (O) while being forced to learn the new one. This incoherent optimization leads to significant parameter shifts that overwrite old knowledge.

Comparison of sequence labeling vs. span-based modeling in CL-NER.

Methodology: The SpanKL Approach

SpanKL addresses these issues by moving from token-level tagging to a span-based framework.

1. Span-Based Architecture

Instead of classifying individual tokens, SpanKL enumerates all possible text spans within a sentence (up to a maximum length). Each span is represented using a contextual encoder, such as BERT.

2. Independent Binary Classification

Rather than a single multi-class classifier, SpanKL employs independent binary classifiers for each entity type. For any given span, the model independently predicts the probability of it belonging to each known entity class. This multi-label loss (Binary Cross Entropy) ensures that learning a new class does not inherently contradict the absence of a previous class.

3. Knowledge Distillation (KD)

To further preserve old knowledge, SpanKL utilizes a teacher-student distillation framework. The model from the previous incremental step acts as a teacher, providing pseudo-labels for old entity types on the new task's data. A Bernoulli KL divergence loss is used to align the current model's predictions with the teacher's for previously learned classes.

Experimental Results

We evaluated SpanKL on the OntoNotes 5.0 and Few-NERD datasets across various data-splitting strategies to ensure robustness.

F1 score performance across incremental steps on OntoNotes and Few-NERD.

Performance and Stability

As shown in the F1 curves, SpanKL maintains significantly higher performance on old tasks compared to baselines like AddNER and ExtendNER. On OntoNotes, SpanKL reduced the performance gap between continual learning and joint training (non-CL) to nearly zero in several setups. In the challenging Few-NERD multi-type scenario, SpanKL outperformed the next best model by approximately 2.8% F1 at the final step.

The stability of SpanKL is particularly noteworthy. While sequence-labeling models show sharp declines in performance for older classes as new ones are added, SpanKL's performance curves remain relatively flat, indicating a high resistance to catastrophic forgetting.

Conclusion

SpanKL demonstrates that the choice of modeling paradigm is crucial for continual learning. By shifting from sequence labeling to independent span classification, we eliminate the primary source of optimization conflict in CL-NER. SpanKL provides a robust and simple baseline for future research in adaptive information extraction.

Summary updated

BibTeX
@inproceedings{zhang2023neural,
  author = {Yunan Zhang and Qingcai Chen},
  title = {{A Neural Span-Based Continual Named Entity Recognition Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26638},
}
Figure