YNUNLP at SemEval-2023 Task 2: The Pseudo Twin Tower Pre-training Model for Chinese Named Entity Recognition

Jing Li, Xiaobing ZhouACL 2023

TL;DR

This paper introduces a novel BERT-CRF-Rdrop model for fine-grained Chinese Named Entity Recognition (NER). By combining BERT's contextual embeddings, CRF's sequence labeling capabilities, and R-drop's regularization, the model achieves competitive results on the challenging MultiCoNER II dataset, demonstrating the effectiveness of this hybrid approach for complex NER tasks.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that aims to identify and categorize named entities (like people, organizations, and locations) within text. While NER is well-established for languages like English, Chinese presents unique challenges. These include the absence of clear word boundaries (requiring careful word segmentation), complex entity structures (nesting, indefinite length), and the proliferation of internet neologisms. The SemEval-2023 Task 2: MultiCoNER II specifically targets fine-grained NER, demanding the identification of a much larger and more nuanced set of entity categories (36 in total for the Chinese track), making it a particularly difficult benchmark. This paper tackles the problem of accurately identifying these fine-grained entities in Chinese text, pushing the boundaries of existing NER systems.

Methods

The authors propose a "Pseudo Twin Tower" model, which they refer to as the BERT-CRF-Rdrop model. This architecture integrates three key components:

  • BERT (Bidirectional Encoder Representations from Transformer): This serves as the backbone for generating rich contextual embeddings for each character in the input Chinese text. BERT's ability to capture deep semantic relationships within sentences is crucial for understanding the nuances of named entities.
  • CRF (Conditional Random Fields): Following BERT's character-level embeddings, a CRF layer is employed. CRF is a powerful sequence labeling model that considers the dependencies between adjacent labels, helping to ensure the coherence and accuracy of the predicted entity sequences. This is particularly useful for NER where the label of one token often influences the labels of its neighbors.
  • R-drop (Regularization based on Dropout): To improve the model's generalization and robustness, the authors incorporate R-drop. This regularization strategy, built upon standard dropout, encourages the model to produce similar output distributions for the same input data when subjected to slightly different dropout masks. This helps to prevent overfitting and enhance the model's ability to handle unseen data.

The model is trained end-to-end, with BERT providing the initial representations, CRF refining the sequence labeling, and R-drop acting as a regularizer during training.

Datasets

The authors utilized the official raw data provided for the SemEval-2023 Task 2: MultiCoNER II (Track 9 - Chinese). This dataset is derived from three sources: Low-Context Wikipedia, MS-MARCO Question, and ORCAS Search Query. The Chinese track of this task specifically involves identifying entities from a fine-grained taxonomy of 36 NE classes. The dataset comprises 9,759 training examples, 506 validation examples, and at least 100,000 final test data points.

Key Insights and Ideas

  • Hybrid Approach for Complex NER: The paper highlights that combining powerful pre-trained language models (like BERT) with sequence labeling models (like CRF) and advanced regularization techniques (like R-drop) is a highly effective strategy for tackling complex NER tasks, especially those with fine-grained categories.
  • R-drop for Improved Generalization: The inclusion of R-drop proved beneficial, suggesting that regularizing the model's output distributions leads to better generalization on unseen data, a critical factor in competitive NLP tasks.
  • Chinese NER Challenges Addressed: The paper implicitly addresses the inherent difficulties of Chinese NER by using BERT for character-level representation, which bypasses the need for explicit word segmentation and its associated error propagation.

Key Results

The BERT-CRF-Rdrop model achieved a macro average F1 score of 0.5657 on the Chinese test set. This performance placed the team 15th out of 22 participating teams in the SemEval-2023 Task 2 Chinese track. The paper also presents a comparison of different model configurations:

  • BERT: Precision 0.5002, Recall 0.5813, F1 0.5421
  • BERT-CRF: Precision 0.5106, Recall 0.6041, F1 0.5552
  • BERT-CRF-Rdrop: Precision 0.5288, Recall 0.6203, F1 0.5657

These results demonstrate that the addition of CRF and R-drop progressively improved the performance, with the full BERT-CRF-Rdrop model yielding the best results.

Limitations and Future Work

Limitations: * The authors mention that due to time and computing resource limitations, they only developed and evaluated the Chinese model. * The paper focuses on a single language track, and the performance on other languages within the MultiCoNER II task is not discussed.

Future Work: The authors suggest that future work will focus on improving performance by developing more complex integration strategies and exploring more diverse models. This could involve incorporating external knowledge bases, exploring different pre-training objectives, or experimenting with more advanced neural architectures.

Summary updated

BibTeX
@inproceedings{li2023ynunlp,
  author = {Jing Li and Xiaobing Zhou},
  title = {{YNUNLP at SemEval-2023 Task 2: The Pseudo Twin Tower Pre-training Model for Chinese Named Entity Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure