The Comparison Between the Tools for Named Entity Recognition

Wenjie ZhangTuwhera 2020

TL;DR

This paper compares the performance of BERT, a powerful pre-trained language model, against a sophisticated BiLSTM-CRF-CNN model for Named Entity Recognition (NER). The study highlights BERT's effectiveness and relative simplicity, while also acknowledging its computational demands and suggesting hybrid approaches for even better results.

What Problem Does This Paper Address?

The paper tackles the challenge of Named Entity Recognition (NER), a crucial task in Natural Language Processing (NLP) that involves identifying and categorizing named entities (like people, organizations, and locations) in text. Traditional methods for NLP tasks, including NER, were often slow and cumbersome, especially when dealing with large datasets. While Machine Learning and Deep Learning have introduced more efficient and accurate approaches, the field is constantly evolving. This research specifically addresses the need to compare the capabilities of a cutting-edge pre-trained model, BERT, against a well-established deep learning architecture (BiLSTM-CRF-CNN) for NER, aiming to understand their relative strengths and weaknesses.

Methods

The study employs a comparative approach, evaluating two primary techniques for NER:

  • BERT (Bidirectional Encoder Representations from Transformers): This is a pre-trained language model developed by Google, known for its powerful contextual understanding of language. The paper focuses on using BERT with fine-tuning for specific tasks, highlighting its "plug-and-play" nature.
  • BiLSTM-CRF-CNN (Bi-directional Long Short-Term Memory - Conditional Random Field - Convolutional Neural Network): This represents a more traditional, yet elaborate, deep learning architecture for sequence labeling tasks like NER. It combines the strengths of Bi-directional LSTMs for capturing long-range dependencies, CNNs for feature extraction, and CRFs for modeling label sequences.

The research involves a literature review to understand the evolution of NLP and deep learning techniques, followed by an experimental comparison of BERT and the BiLSTM-CRF-CNN model on NER tasks.

Datasets

The primary dataset used for training and evaluation in this experiment is:

  • CoNLL 2003: This is a widely recognized dataset specifically designed for NER tasks, particularly for English. The paper mentions using its training, validation, and test files.
  • Reuters Corpus: This corpus, consisting of Reuters news stories from August 1996 to August 1997, was used as the English corpus for the experiment.

Key Insights and Ideas

  • BERT's Power and Simplicity: The research underscores BERT's significant impact on NLP, demonstrating its ability to achieve high performance with relatively straightforward fine-tuning, making it accessible for researchers and developers.
  • Computational Demands of BERT: A key limitation highlighted is BERT's substantial computational requirements, particularly for its larger versions, which necessitate powerful GPUs and significant RAM for effective training.
  • Hybrid Approaches for Superior Performance: The study suggests that combining BERT with other techniques like CRF or BiLSTM-CRF can lead to even more remarkable accuracy, indicating that while BERT is powerful, it can be further enhanced.
  • The Evolving Landscape of Pre-trained Models: The paper touches upon the trend of pre-trained models becoming increasingly prevalent and the ongoing development of more efficient versions (like ALBERT), suggesting a future where model size isn't the sole determinant of performance.

Key Results

The experimental results, presented in Table 1, show:

  • Overall Performance: The BiLSTM-CNN-CRF model achieved a slightly higher overall F1 score (90.76%) compared to the pure BERT model (90.55%).
  • Label-Specific Performance (BERT): BERT demonstrated strong performance across most labels, with particularly high scores for PER (Person) and LOC (Location). However, its performance on MISC (Miscellaneous) entities was notably lower, likely due to the smaller number of MISC entities in the dataset.
  • Training Loss: The training loss curve for BERT on the CoNLL 2003 dataset showed a rapid and smooth decrease, indicating the model's effectiveness in learning and its power in handling NER tasks.

Limitations and Future Work

  • Computational Limitations: The most significant limitation acknowledged is the hardware configuration used for the experiment. The available GPU (8GB RAM) was insufficient to run the larger, more powerful versions of BERT, thus restricting the experiment to the "base" version. This prevented the reproduction of results that might have been achieved with more substantial computational resources.
  • Future Research Directions:
    • Optimizing BERT for Resource-Constrained Environments: The authors suggest that exploring ways to optimize BERT for training on GPUs with limited RAM is a crucial area for future research.
  • Hybrid Models: Further investigation into BERT-CRF and BERT-BILSTM-CRF models is recommended for achieving even higher accuracy.
  • Exploring Model Size vs. Performance: The development of models like ALBERT suggests that future research should focus on efficient model architectures and parameter utilization rather than solely on increasing model size.
  • Applications on Mobile Devices: The paper hints at the potential for BERT applications on mobile phones as a probable direction for future research.

Summary updated

Abstract

NLP (Natural language processing) is currently been wildly using in our modern daily life, such as spam email detecting, prediction of potential criminals by analysing the provided information, sentiment analysing and many so on. In the old-time, the popular way to solve NLP tasks can be done by hands or by computers with some strictly given disciplines, these ways are tiresome and often too slow if we have to deal with a huge amount of data or information. Until now, some of the companies and researchers are still using these ways to do their jobs. Besides that, thanks to the development of the computer hardware, we have some relatively new, prevalent and highly accurate ways to help us to finish these tasks, from Machine Learning (such as SVM----supported vector machines) to DNN (Deep Neural Networks) and pretrained models(like ELMO, OpenAI GPT), therefore in modern data companies and online shopping enterprises, the previous ways are gradually decreasing in use to do these tasks because they are obsolete and not suitable to be used in the data era anymore. But now, we have the brand-new approach to cope with the NLP tasks, which is called BERT, it is the abbreviation of Bidirectional Encoder Representations from Transformer. The BERT was introduced and inaugurated by the Google development team in late 2018, it is a pretrained language model for multiple language processing tasks, and also for multiple languages by using the unsupervised learning way. This model is powerful for doing the realistic language processing tasks and it has already broken 11 records in the NLP area in 2018. And, it is simple to use, it just needs fine-tuning and only one additional layer for output without other specific architectures for some specific tasks (such as labelling, classification or question answering). More information of this new technique will be introduced in a detailed way. The method used for completing this thesis is based on literature review, and a comparison between BERT and one prevailing and elaborate DNN technique which is called BILSTM-CRF-CNN (Bi-directional Long Short-Term Memory-Conditional Random Field-Convolutional Neural Networks) in dealing with Name Entity Recognition (NER) and the results of the comparison will be shown. Also, my opinions about the BERT, the suggestions, and potential future issues of future researches are given.
BibTeX
@inproceedings{zhang2020comparison,
  author = {Wenjie Zhang},
  title = {{The Comparison Between the Tools for Named Entity Recognition}},
  booktitle = {Tuwhera},
  year = {2020},
}
Figure