A More Efficient Chinese Named Entity Recognition base on BERT and Syntactic Analysis

Xiao Fu, Guijun Zhang

Abstract

We propose a new Named entity recognition (NER) method to effectively make use of the results of Part-of-speech (POS) tagging, Chinese word segmentation (CWS) and parsing while avoiding NER error caused by POS tagging error. This paper first uses Stanford natural language process (NLP) tool to annotate large-scale untagged data so as to reduce the dependence on the tagged data; then a new NLP model, g-BERT model, is designed to compress Bidirectional Encoder Representations from Transformers (BERT) model in order to reduce calculation quantity; finally, the model is evaluated based on Chinese NER dataset. The experimental results show that the calculation quantity in g-BERT model is reduced by 60% and performance improves by 2% with Test F1 to 96.5 compared with that in BERT model.

Introduction

Named Entity Recognition (NER) in Chinese presents unique challenges compared to English, primarily due to the lack of explicit word boundaries. Traditional pipelines require Chinese Word Segmentation (CWS) and Part-of-Speech (POS) tagging as preprocessing steps. This serial approach often leads to error propagation, where inaccuracies in segmentation directly degrade NER performance. Furthermore, state-of-the-art models like BERT-base are computationally intensive, with approximately 110 million parameters, making them difficult to deploy in resource-constrained environments. This paper introduces g-BERT, a syntax-aware, compressed architecture designed to mitigate these issues by integrating syntactic analysis directly into the model's learning process.

The g-BERT Architecture

The core innovation of g-BERT is the introduction of a "gate node" (or global node) mechanism within the Transformer layers. In standard BERT, every token attends to every other token, which is computationally expensive. In g-BERT, a global node is added to each layer to perform max-pooling across the sentence, capturing essential contextual information. A gating mechanism then regulates how this global context is distributed back to the network. This architectural shift allows the model to maintain high performance while significantly reducing the number of hidden layers and attention heads.

Comparison between standard BERT and the proposed g-BERT architecture, highlighting the gate node mechanism.

By utilizing these gate nodes, g-BERT achieves a more efficient representation of local and global token relationships, enabling a reduction in model size to approximately 30-40 million parameters without sacrificing accuracy.

Multi-task Pre-training and "Silver" Labels

To address the scarcity of high-quality labeled Chinese NER data, the authors employed a data augmentation strategy using "silver" labels. Large-scale untagged Wikipedia data was automatically annotated using the Stanford NLP toolkit for CWS, POS, and syntactic parsing.

Rather than relying solely on Masked Language Modeling (MLM), g-BERT is trained using a multi-task objective. The model is forced to simultaneously predict word boundaries and POS tags alongside its primary objectives. This ensures that the model develops an inherent understanding of Chinese syntax during the pre-training phase, which it then leverages during fine-tuning for NER.

The g-BERT multi-task framework, showing the shared encoder feeding into specialized CWS and NER CRF heads.

As shown in the framework, the g-BERT encoder provides representations to both a CWS Conditional Random Field (CRF) and an NER CRF. This parallel processing eliminates the serial bottleneck of traditional pipelines and reduces the impact of segmentation errors.

Experimental Results

The model was evaluated on the SIGHAN 2006 NER dataset (Sighan-NER), focusing on Person, Location, and Organization entities.

Key findings include:

  • Performance: g-BERT achieved a state-of-the-art F1 score of 96.5%, surpassing BERT-base (92.6%) and ERNIE (93.8%).
  • Efficiency: The computational overhead was reduced by 60% compared to the BERT baseline.
  • Scalability: With a parameter count of 30M-40M, g-BERT is significantly more suitable for real-time applications and edge deployment than the 110M-parameter BERT-base.

Conclusion

g-BERT demonstrates that integrating syntactic awareness and global gating mechanisms can lead to more efficient and accurate NLP models. By baking Chinese linguistic structure into the pre-training phase and utilizing a compressed architecture, the authors successfully addressed the dual challenges of error propagation and computational cost in Chinese NER. Future research may focus on refining the automated labeling process to further improve the quality of "silver" training data and testing the model's robustness on less formal text sources like social media.

Summary updated

Abstract

We propose a new Named entity recognition (NER) method to effectively make use of the results of Part-of-speech (POS) tagging, Chinese word segmentation (CWS) and parsing while avoiding NER error caused by POS tagging error. This paper first uses Stanford natural language process (NLP) tool to annotate large-scale untagged data so as to reduce the dependence on the tagged data; then a new NLP model, g-BERT model, is designed to compress Bidirectional Encoder Representations from Transformers (BERT) model in order to reduce calculation quantity; finally, the model is evaluated based on Chinese NER dataset. The experimental results show that the calculation quantity in g-BERT model is reduced by 60% and performance improves by 2% with Test F1 to 96.5 compared with that in BERT model.
BibTeX
@misc{fu2021more,
  author = {Xiao Fu and Guijun Zhang},
  title = {{A More Efficient Chinese Named Entity Recognition base on BERT and Syntactic Analysis}},
  year = {2021},
  eprint = {2101.11423},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.11423},
}
Figure