Application of Pre-training Models in Named Entity Recognition

Yu Wang, Yining Sun, Zuchang Ma, Lisheng Gao, Yang Xu, Ting Sun

TL;DR

This paper explores how four prominent pre-trained language models (BERT, ERNIE, ERNIE2.0-tiny, and RoBERTa) perform when fine-tuned for Named Entity Recognition (NER). They found that RoBERTa, with its dynamic masking and removal of the Next Sentence Prediction task, achieved state-of-the-art results on the MSRA-2006 dataset, suggesting that sentence-level pre-training tasks might not be optimal for NER.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a crucial task in Natural Language Processing (NLP) that involves identifying and classifying named entities (like people, organizations, and locations) within text. Traditional methods relied on machine learning or deep learning, but these often required extensive labeled data, which is costly and time-consuming to create. The advent of large pre-trained language models, which learn rich semantic and syntactic representations from vast amounts of unlabeled text, offered a promising avenue to improve NER performance without relying solely on labeled data. This paper investigates which of these pre-trained models, and their specific pre-training strategies, are most effective for the NER task.

Methods

The authors first introduce and describe the architectures and pre-training tasks of four popular pre-trained models:

  • BERT (Bidirectional Encoder Representations from Transformers): Uses a multi-layer bidirectional Transformer architecture and is pre-trained on two tasks: Masked Language Model (MLM) and Next Sentence Prediction (NSP).
  • ERNIE (Enhanced Representation through Knowledge Integration): Shares BERT's architecture but employs entity-level and phrase-level masking strategies in addition to basic masking, aiming to capture knowledge more effectively. It also includes a Dialogue Language Model (DLM) task.
  • ERNIE2.0-tiny: A compressed version of ERNIE 2.0, featuring a smaller Transformer architecture (3 layers) but with a focus on continual pre-training across various tasks (word-aware, structure-aware, semantic-aware).
  • RoBERTa (A Robustly Optimized BERT Pretraining Approach): Similar to BERT in architecture but modifies the masking strategy to be dynamic (changing masking positions each epoch) and removes the NSP task.

For applying these models to NER, the researchers adopted a fine-tuning approach. They added a fully connected (FC) layer and a Conditional Random Field (CRF) layer on top of the pre-trained models. The FC layer helps extract higher-level representations, while the CRF layer captures the dependencies between output tags, which is crucial for sequence labeling tasks like NER.

Datasets

The primary dataset used for evaluation in this paper is the MSRA-2006 dataset, a benchmark for Chinese Named Entity Recognition. The paper does not introduce any new datasets.

Key Insights and Ideas

  • Pre-trained models significantly boost NER performance: The experiments clearly show that fine-tuning pre-trained models leads to substantial improvements in NER performance compared to a baseline BiGRU+CRF model.
  • Model depth matters: ERNIE2.0-tiny, despite its advanced continual pre-training, underperformed due to its significantly shallower architecture (3 Transformer layers vs. 12 in others). This suggests that deeper models are better at capturing the complex linguistic features needed for NER.
  • Sentence-level pre-training might not be ideal for NER: RoBERTa's superior performance, achieved by removing the NSP task (which focuses on sentence relationships), suggests that NER tasks may not benefit from, and could even be hindered by, pre-training objectives that focus on inter-sentence dependencies.
  • Dynamic masking is advantageous: RoBERTa's dynamic masking strategy, which varies the masked tokens across epochs, appears to help the model adapt better and learn richer representations compared to BERT and ERNIE's static masking.
  • Potential issues with ERNIE's masking: The slightly lower performance of ERNIE compared to BERT, despite its more sophisticated masking, might be due to segmentation errors introduced by its entity-level and phrase-level masking strategies.

Key Results

The paper reports the following F1 scores on the MSRA-2006 dataset:

  • Baseline (BiGRU+CRF): 90.32%
  • BERT-base: 93.30%
  • ERNIE: 93.37%
  • ERNIE-tiny: 86.52%
  • RoBERTa: 94.17%

RoBERTa achieved the highest F1 score, outperforming BERT and ERNIE by a notable margin. ERNIE-tiny performed significantly worse than the baseline, highlighting the importance of model depth.

Limitations and Future Work

Limitations: * The study focused on a single NER dataset (MSRA-2006), which might limit the generalizability of the findings to other languages or domains. * The authors infer potential issues with ERNIE's masking strategy but do not provide direct experimental evidence to confirm this.

Future Work: The authors suggest that investigating how model structures can be adapted for different downstream tasks, including NER, would be a valuable direction for future research. This implies exploring task-specific architectural modifications or further refinements to pre-training objectives.

Summary updated

Abstract

Named Entity Recognition (NER) is a fundamental Natural Language Processing (NLP) task to extract entities from unstructured data. The previous methods for NER were based on machine learning or deep learning. Recently, pre-training models have significantly improved performance on multiple NLP tasks. In this paper, firstly, we introduce the architecture and pre-training tasks of four common pre-training models: BERT, ERNIE, ERNIE2.0-tiny, and RoBERTa. Then, we apply these pre-training models to a NER task by fine-tuning, and compare the effects of the different model architecture and pre-training tasks on the NER task. The experiment results showed that RoBERTa achieved state-of-the-art results on the MSRA-2006 dataset.
BibTeX
@misc{wang2020application,
  author = {Yu Wang and Yining Sun and Zuchang Ma and Lisheng Gao and Yang Xu and Ting Sun},
  title = {{Application of Pre-training Models in Named Entity Recognition}},
  year = {2020},
  eprint = {2002.08902},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.08902},
}
Figure