Janko at SemEval-2023 Task 2: Bidirectional LSTM Model Based on Pre-training for Chinese Named Entity Recognition

Jiankuo Li, Zhengyi Guan, Haiyan DingACL 2023

TL;DR

This paper presents the Janko team's approach to the SemEval-2023 Task 2 for Chinese Named Entity Recognition (NER). They developed a BERT-BiLSTM-RDrop model, leveraging pre-trained BERT embeddings, a bidirectional LSTM for context, and R-Drop for regularization, achieving a macro-averaged F1 score of 0.579.

What Problem Does This Paper Address?

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) focused on identifying and categorizing entities like people, organizations, and locations within text. While NER is crucial for applications like information extraction and question answering, it's challenging due to word ambiguity (e.g., "Washington" as a place or person) and vague entity boundaries. The SemEval-2023 Task 2, specifically the MultiCoNER 2 challenge, aimed to push NER capabilities further by introducing a more granular set of entity tags (over 30) and shorter, potentially noisier text segments, making the task more complex and realistic than previous iterations. The Chinese track, in particular, presents unique challenges due to the absence of spaces between words, making word segmentation a prerequisite for accurate NER.

Methods

The Janko team's approach is built upon a three-part architecture:

  • BERT Embeddings: They utilize the pre-trained BERT (Bidirectional Encoder Representations from Transformers) model, specifically BERT-base-chinese, to generate contextualized word embeddings. BERT's transformer architecture and its pre-training tasks (Masked Language Modeling and Next Sentence Prediction) allow it to capture rich semantic information. The last layer of BERT is used as the input embedding for the subsequent layers.
  • BiLSTM Neural Network: Following BERT, a Bidirectional Long Short-Term Memory (BiLSTM) network is employed. BiLSTM is well-suited for sequential data like text, as it processes information in both forward and backward directions. This allows the model to capture dependencies from both preceding and succeeding words, which is crucial for understanding the context of named entities.
  • R-Drop Regularization: To enhance the model's robustness and mitigate overfitting, the researchers incorporate R-Drop. R-Drop is a regularization technique that aims to minimize the Kullback-Leibler (KL) divergence between two different output distributions generated by applying dropout twice to the same input. This helps to regularize the model's predictions during training and testing.

The final model is thus named BERT-BiLSTM-RDrop.

Datasets

The primary dataset used in this work is from the SemEval-2023 Task 2: MultiCoNER 2 challenge. The authors specifically participated in the Chinese track. The dataset consists of training, validation, and test sets. The training and validation sets provide labels for named entities, while the test set contains only text. The organizers divided the competition into 13 tracks based on language.

Key Insights and Ideas

  • Pre-trained Embeddings are Key: The success of the BERT model in capturing rich linguistic features is highlighted, demonstrating its effectiveness as a foundation for downstream NLP tasks like NER.
  • Bidirectional Context Matters: The BiLSTM layer's ability to process information in both directions significantly improves the model's understanding of semantic dependencies, leading to better entity recognition.
  • R-Drop for Robustness: R-Drop proves to be a valuable regularization technique, helping to compensate for the inconsistencies introduced by dropout during training and testing, thereby improving the model's overall robustness.
  • Chinese NER Challenges: The paper implicitly acknowledges the inherent difficulties of Chinese NER, such as the lack of explicit word boundaries, which necessitates careful handling of word segmentation and contextual understanding.

Key Results

The Janko team's BERT-BiLSTM-RDrop model achieved a macro-averaged F1 score of 0.579 on the test set for the Chinese track of SemEval-2023 Task 2. This performance placed them 14th out of 21 participating teams. Ablation studies showed that the BiLSTM component provided a significant improvement over a BERT-only model. While R-Drop offered limited improvement, it was expected to enhance robustness. The paper also notes that other pre-trained models like RoBERTa-base and XLM-ROBERTa-base did not perform as well as BERT-base-chinese on their validation set.

Limitations and Future Work

The authors acknowledge that the improvement from R-Drop was limited, as it primarily reinforces the robustness to dropout rather than introducing entirely new capabilities. They also mention that integrating CNNs with BiLSTM or using voting methods did not yield significant improvements in their experiments.

For future work, the authors plan to:

  • Continue improving the system.
  • Explore more advanced models.
  • Investigate model integration methods.

Summary updated

BibTeX
@inproceedings{li2023janko,
  author = {Jiankuo Li and Zhengyi Guan and Haiyan Ding},
  title = {{Janko at SemEval-2023 Task 2: Bidirectional LSTM Model Based on Pre-training for Chinese Named Entity Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure