TL;DR
This paper presents the NCUEE-NLP system for Chinese Named Entity Recognition (NER) in the SemEval-2022 MultiCoNER task. They achieved a strong result by combining BERT embeddings with a BiLSTM-CRF model, demonstrating the effectiveness of character-level analysis for complex Chinese NER.
What Problem Does This Paper Address?
Named Entity Recognition (NER) is a crucial task in information extraction, aiming to identify and categorize entities like people, organizations, and locations within text. However, recognizing "complex" named entities, such as titles of creative works, presents a significant challenge due to their varied linguistic forms and potential for semantic ambiguity. This is particularly true for Chinese, which lacks capitalization and word delimiters, making word segmentation a prerequisite and a potential source of errors. The SemEval-2022 Task 11 (MultiCoNER) specifically targeted this challenge by focusing on multilingual, complex NER in short, low-context settings across 11 languages, including Chinese. The NCUEE-NLP team focused on the Chinese track (Track 9), aiming to build a robust system for this difficult task.
Methods
The NCUEE-NLP system employs a three-part architecture:
- BERT Embeddings: The system leverages BERT (Bidirectional Encoder Representations from Transformer) to generate contextual embeddings for each character in the input Chinese sentences. BERT's ability to capture context is crucial, especially since incorrect word segmentation in Chinese can propagate errors. They specifically use the last layer of BERT to obtain these character-level embeddings.
- Bidirectional LSTM Networks (BiLSTM): Following the BERT embeddings, a BiLSTM layer is used. LSTMs are recurrent neural networks adept at processing sequential data. By using a bidirectional LSTM, the model can capture information from both the left-to-right and right-to-left directions of the character sequence, providing a richer understanding of context.
- CRF Sequence Labeling: Finally, a Conditional Random Field (CRF) layer is applied on top of the BiLSTM outputs. The CRF layer learns to predict the sequence of labels for each character, taking into account the dependencies between adjacent labels. This is crucial for accurately identifying the boundaries and types of named entities. The system uses the standard BIO (Beginning, Inside, Outside) tagging scheme.
Datasets
The authors utilized several datasets for training and evaluation:
- Official Datasets: Provided by the SemEval-2022 Task 11 organizers, these included training, validation, and test sets.
- External Datasets: To augment the official data and improve robustness, they also incorporated external datasets:
- MSRA (Levow, 2006)
- Weibo (Peng and Dredze, 2015)
- People Daily (PD)
- Boson
- CLUENER (Xu et al., 2020)
- LG
It's noted that the external datasets primarily contained annotations for Person (PER) and Location (LOC) entities, while CLUENER and LG were adapted to provide instances for the Creative Work (CW) category. The authors did not find suitable external data for Group (GRP), Corporation (CORP), and Product (PROD) entities.
Key Insights and Ideas
- Character-level BERT is Key for Chinese NER: The paper strongly advocates for character-level processing with BERT for Chinese NER. This approach circumvents the challenges of word segmentation, which is a known bottleneck and error source in Chinese NLP.
- BiLSTM-CRF Remains a Strong Baseline: Despite the rise of more complex architectures, the combination of BiLSTM and CRF, when paired with powerful pre-trained embeddings like BERT, continues to yield competitive results for sequence labeling tasks like NER.
- External Data Doesn't Always Help: Surprisingly, augmenting the official training data with external datasets did not consistently improve the overall F1 performance of the BiLSTM-CRF model. This suggests that the quality and domain relevance of the training data are paramount.
- Entity Definition Matters for Performance: The authors observed that entity categories with clearer definitions (like LOC and PER) achieved better F1 scores. Categories like GRP, CORP, and CW, which can be more ambiguous and harder to distinguish even for human annotators, presented greater challenges.
Key Results
The NCUEE-NLP system achieved a macro-averaging F1-score of 0.7418 on the test set. This performance ranked them seventh out of 21 participating teams in the Chinese track of SemEval-2022 Task 11. The best performance was achieved using the BERT-BiLSTM-CRF model trained solely on the official data. Detailed results per entity class showed that LOC and PER entities were recognized with the highest F1 scores.
Limitations and Future Work
- External Data Impact: The paper notes that incorporating external data did not lead to improved F1 scores, suggesting that further investigation into data selection and integration strategies might be beneficial.
- Ambiguous Entity Categories: The authors acknowledge that distinguishing between certain entity categories (e.g., GRP/CORP and PROD/CW) is inherently difficult, even with manual annotation. Future work could explore more sophisticated methods for disambiguation or refine entity definitions.
- Hyperparameter Tuning: While hyperparameters were optimized, further fine-tuning could potentially yield marginal improvements.
- Exploring Other Embeddings: Although RoBERTa and MacBERT were tested, they did not outperform BERT. Future work could explore other advanced pre-trained models or novel embedding fusion techniques.
Summary updated