An Instance Transfer based Approach Using Enhanced Recurrent Neural Network for Domain Named Entity Recognition

Lin Li, Yueqing SunarXiv 2018 · arXiv

Abstract

Named Entity Recognition (NER) is a critical component of information extraction, yet its performance often degrades when applied to niche domains with scarce labeled data. This paper presents a novel domain adaptation framework that combines a similarity-aware instance transfer strategy with an Enhanced Recurrent Neural Network (ERNN). By utilizing kernel-based instance weighting and a hybrid activation function designed to mitigate gradient saturation, our approach effectively leverages source domain knowledge. Furthermore, a co-training mechanism with Conditional Random Fields (CRF) allows the model to exploit unlabeled target data. Evaluations on a specialized Chinese politics dataset show a substantial F1-score increase from 0.805 to 0.933, demonstrating the robustness of the proposed method in "cold start" scenarios.

Introduction

Modern NER systems typically rely on large-scale labeled datasets, which are often unavailable for specialized domains such as high school politics textbooks. Manual annotation is prohibitively expensive and time-consuming. This research addresses the "Domain Adaptation" problem: how to effectively transfer knowledge from a well-resourced source domain (e.g., general news) to a target domain with little to no labeled data.

Methodology

Instance Transfer Strategy

To minimize noise from irrelevant source data, we employ a similarity-aware instance transfer strategy. We use Gaussian Radial Basis Function (RBF) and Polynomial kernels to compute the similarity between source domain sentences and the target domain. This allows the model to prioritize and weight source instances that are most representative of the target domain's linguistic patterns.

Enhanced Recurrent Neural Network (ERNN)

The core of our architecture is the ERNN, which introduces two significant improvements over standard RNNs:

  1. Hybrid Activation Function: Standard Sigmoid functions suffer from saturation, leading to vanishing gradients. We developed a hybrid function that blends the Sigmoid with a linear approximation. This maintains the probabilistic interpretation of the output while ensuring a steady gradient flow during backpropagation.
The proposed hybrid activation function compared to a standard Sigmoid, showing the linear approximation used to prevent saturation.
  1. Confluent Layer Architecture: We modified the standard RNN structure by adding a "confluent layer." This additional layer (represented as the $S$ layer in Figure 3) specifically integrates knowledge from transferred source instances into the hidden state transitions, allowing for a more nuanced representation of domain-specific features.
Unrolled architecture of the Enhanced RNN (ERNN) showing the dual-layer structure ($T$ and $S$) designed for domain adaptation.

Co-training Framework

To further enhance performance using unlabeled target data, we implement a semi-supervised co-training approach. The ERNN and a traditional CRF model serve as two independent learners. They iteratively label unlabeled data and add the most confident predictions to the training set, effectively "teaching" each other and refining the decision boundaries.

Experimental Setup

The model was evaluated using the People’s Daily (1998) corpus as the source domain and a custom Chinese High School Politics Text dataset as the target domain. The target dataset consisted of over 15,000 sentences, with only 3,043 manually labeled for validation and testing.

Results and Discussion

Our experiments yielded several key insights:

  • Superiority of Deep Learning Baselines: On standard benchmarks like ATIS, RNN-based models consistently outperformed HMM and CRF baselines.
  • Impact of Instance Transfer: In a zero-label target scenario, the ERNN with instance transfer achieved an F1-score of 0.9328, a significant jump from the 0.8052 achieved by a standard RNN using raw source data.
  • Effectiveness of Co-training: The integration of unlabeled data via co-training pushed the final F1-score to a peak of 0.9402. However, we observed that the CRF's lower initial accuracy can eventually introduce noise, leading to slight performance degradation in later iterations.

Conclusion

This paper demonstrates that combining intelligent instance selection with architectural enhancements like the ERNN can overcome the data scarcity challenges in domain-specific NER. Future work will focus on extending this framework to relationship extraction to support more complex downstream tasks like automated Question & Answer systems.

Summary updated

Abstract

Recently, neural networks have shown promising results for named entity recognition (NER), which needs a number of labeled data to for model training. When meeting a new domain (target domain) for NER, there is no or a few labeled data, which makes domain NER much more difficult. As NER has been researched for a long time, some similar domain already has well labelled data (source domain). Therefore, in this paper, we focus on domain NER by studying how to utilize the labelled data from such similar source domain for the new target domain. We design a kernel function based instance transfer strategy by getting similar labelled sentences from a source domain. Moreover, we propose an enhanced recurrent neural network (ERNN) by adding an additional layer that combines the source domain labelled data into traditional RNN structure. Comprehensive experiments are conducted on two datasets. The comparison results among HMM, CRF and RNN show that RNN performs bette than others. When there is no labelled data in domain target, compared to directly using the source domain labelled data without selecting transferred instances, our enhanced RNN approach gets improvement from 0.8052 to 0.9328 in terms of F1 measure.
BibTeX
@misc{li2018instance,
  author = {Lin Li and Yueqing Sun},
  title = {{An Instance Transfer based Approach Using Enhanced Recurrent Neural Network for Domain Named Entity Recognition}},
  year = {2018},
  eprint = {1810.06644},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.06644},
}
Figure