Transfer Learning for Sequence Labeling Using Source Model and Target Data

Lingzhen Chen, Alessandro MoschittiAAAI 2019 · DOI

Abstract

This paper presents a progressive learning framework for sequence labeling tasks, such as Named Entity Recognition (NER), designed to adapt existing models to new entity categories without requiring access to original source training data. The proposed approach utilizes parameter transfer and a novel "neural adapter"—a secondary Bidirectional Long Short-Term Memory (BLSTM) network—to bridge source and target models. Experimental results on the CoNLL 2003 and I-CAB datasets demonstrate that this method significantly improves performance on new labels, particularly in data-scarce scenarios, by effectively resolving label disagreements between source and target domains.

Introduction

In traditional Named Entity Recognition (NER), models are typically trained on a static set of categories (e.g., Person, Location, Organization). However, real-world applications are dynamic, frequently requiring the inclusion of new entity types as they emerge. Standard approaches, such as retraining from scratch or extensive manual annotation, are often impractical due to high computational costs or the unavailability of original source data due to privacy and copyright constraints. This research addresses the challenge of teaching an existing model new categories using limited target data while ensuring it retains its prior knowledge.

Methodology

The authors build upon the state-of-the-art architecture for sequence labeling: a Bidirectional LSTM (BLSTM) with Conditional Random Fields (CRF), utilizing both word and character-level embeddings. They propose two primary transfer learning (TL) strategies:

  1. Parameter Transfer & Expansion: To accommodate new labels, the output layer of the source model ($M_S$) is "stretched" by adding new neurons. Weights for existing categories are copied over, while weights for new categories are initialized based on the mean and standard deviation of the pre-trained weights. The entire model is then fine-tuned on the target data ($D_T$).
  2. The Neural Adapter: The core innovation is a neural adapter designed to resolve "label disagreement." This occurs when an entity belonging to a new category in the target data was previously labeled as "Other" (O) in the source model. The adapter is a separate BLSTM that processes hidden activations from the frozen source model and feeds them into the target model, acting as a learned mapping between the source representations and the new label distribution.
Architecture of the proposed Neural Adapter connecting the frozen source model to the target model.

Experimental Evaluation

The framework was evaluated using the CoNLL 2003 (English) and I-CAB (Italian) benchmarks, modified to simulate a progressive learning environment. To simulate the emergence of new labels, specific categories (e.g., "Location") were withheld during the source training phase and introduced as the "new" category during the transfer learning phase. This setup tests the model's ability to "unlearn" the association of a new category with the "null" label from the source model.

Key Results and Insights

The neural adapter demonstrated substantial performance gains across different languages and entity types. On the Italian I-CAB dataset, the transfer method with the neural adapter improved the F1 score by 10.30 points over a baseline model trained only on the target data.

One of the most significant findings was the adapter's data efficiency. In experiments where target data was limited to 10% or 25% of the full set, the adapter-equipped model converged faster and achieved much higher accuracy than standard fine-tuning.

Learning curves showing Test F1 scores across different percentages of target data (10%, 25%, 50%, 100%), highlighting the efficiency of the Adapter method.

Critical Observations

  • Resolving Conflicts: The study highlights that transfer learning for new labels is primarily about resolving conflicts. The target model must effectively "unlearn" the source model's classification of a new category as "Nothing (O)."
  • Contextual Mapping: The choice of a BLSTM for the adapter allows it to utilize surrounding context to decide if the source model's prediction should be overridden.
  • Fine-tuning vs. Freezing: The authors found that freezing early layers of the model can hinder performance. Allowing all parameters to update enables the model to better recover from labeling disagreements between the source and target datasets.

Conclusion and Future Work

The proposed progressive learning framework offers a robust solution for expanding NER systems to new categories without requiring source data access. While the results are promising, future research could explore how the adapter handles massive expansions (e.g., adding dozens of categories at once) and how it performs under significant domain shifts where the underlying language distribution changes.

Summary updated

BibTeX
@inproceedings{chen2019transfer,
  author = {Lingzhen Chen and Alessandro Moschitti},
  title = {{Transfer Learning for Sequence Labeling Using Source Model and Target Data}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33016260},
}
Figure