Abstract
This paper presents a progressive learning framework for sequence labeling tasks, such as Named Entity Recognition (NER), designed to adapt existing models to new entity categories without requiring access to original source training data. The proposed approach utilizes parameter transfer and a novel "neural adapter"—a secondary Bidirectional Long Short-Term Memory (BLSTM) network—to bridge source and target models. Experimental results on the CoNLL 2003 and I-CAB datasets demonstrate that this method significantly improves performance on new labels, particularly in data-scarce scenarios, by effectively resolving label disagreements between source and target domains.
Introduction
In traditional Named Entity Recognition (NER), models are typically trained on a static set of categories (e.g., Person, Location, Organization). However, real-world applications are dynamic, frequently requiring the inclusion of new entity types as they emerge. Standard approaches, such as retraining from scratch or extensive manual annotation, are often impractical due to high computational costs or the unavailability of original source data due to privacy and copyright constraints. This research addresses the challenge of teaching an existing model new categories using limited target data while ensuring it retains its prior knowledge.
Methodology
The authors build upon the state-of-the-art architecture for sequence labeling: a Bidirectional LSTM (BLSTM) with Conditional Random Fields (CRF), utilizing both word and character-level embeddings. They propose two primary transfer learning (TL) strategies:
- Parameter Transfer & Expansion: To accommodate new labels, the output layer of the source model ($M_S$) is "stretched" by adding new neurons. Weights for existing categories are copied over, while weights for new categories are initialized based on the mean and standard deviation of the pre-trained weights. The entire model is then fine-tuned on the target data ($D_T$).
- The Neural Adapter: The core innovation is a neural adapter designed to resolve "label disagreement." This occurs when an entity belonging to a new category in the target data was previously labeled as "Other" (O) in the source model. The adapter is a separate BLSTM that processes hidden activations from the frozen source model and feeds them into the target model, acting as a learned mapping between the source representations and the new label distribution.
Experimental Evaluation
The framework was evaluated using the CoNLL 2003 (English) and I-CAB (Italian) benchmarks, modified to simulate a progressive learning environment. To simulate the emergence of new labels, specific categories (e.g., "Location") were withheld during the source training phase and introduced as the "new" category during the transfer learning phase. This setup tests the model's ability to "unlearn" the association of a new category with the "null" label from the source model.
Key Results and Insights
The neural adapter demonstrated substantial performance gains across different languages and entity types. On the Italian I-CAB dataset, the transfer method with the neural adapter improved the F1 score by 10.30 points over a baseline model trained only on the target data.
One of the most significant findings was the adapter's data efficiency. In experiments where target data was limited to 10% or 25% of the full set, the adapter-equipped model converged faster and achieved much higher accuracy than standard fine-tuning.
Critical Observations
- Resolving Conflicts: The study highlights that transfer learning for new labels is primarily about resolving conflicts. The target model must effectively "unlearn" the source model's classification of a new category as "Nothing (O)."
- Contextual Mapping: The choice of a BLSTM for the adapter allows it to utilize surrounding context to decide if the source model's prediction should be overridden.
- Fine-tuning vs. Freezing: The authors found that freezing early layers of the model can hinder performance. Allowing all parameters to update enables the model to better recover from labeling disagreements between the source and target datasets.
Conclusion and Future Work
The proposed progressive learning framework offers a robust solution for expanding NER systems to new categories without requiring source data access. While the results are promising, future research could explore how the adapter handles massive expansions (e.g., adding dozens of categories at once) and how it performs under significant domain shifts where the underlying language distribution changes.
Summary updated