TL;DR
This paper introduces a novel Named Entity Recognition (NER) system for ancient Chinese texts, combining the powerful GujiRoBERTa pre-trained language model with LSTM and CRF layers. The proposed system achieves state-of-the-art performance on the EvaHan2025 competition datasets, demonstrating its effectiveness in handling the unique challenges of historical Chinese text.
What Problem Does This Paper Address?
Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that involves identifying and classifying entities like people, locations, and organizations within text. While NER has been extensively studied for modern languages, applying it to ancient Chinese texts presents significant challenges. These include a lack of standardized punctuation, polysemous characters, and complex syntactic structures. Existing methods often struggle with these complexities, especially when dealing with historical documents. This paper aims to bridge this gap by developing a robust NER system specifically tailored for ancient Chinese.
Methods
The core of the proposed system is the GujiRoBERTa model, a pre-trained language model specifically optimized for ancient Chinese. This model serves as the encoder, capturing contextual representations of the input text. To further enhance its capabilities, the authors integrate two key components:
- Long Short-Term Memory (LSTM) Network: An LSTM layer is added to refine the contextual representations from GujiRoBERTa. This helps in capturing sequential dependencies within the text, which is crucial for understanding the flow and relationships between words.
- Conditional Random Fields (CRF): A CRF layer is employed for structured prediction. CRFs are effective in enforcing global label consistency, meaning they consider the relationships between adjacent labels to make more accurate predictions. This is particularly beneficial for NER tasks where the sequence of tags (e.g., "Begin Person," "Inside Person," "End Person") matters.
The overall architecture can be visualized as GujiRoBERTa feeding into an LSTM, which then passes its output through fully connected layers to a CRF layer for final sequence labeling. The model is trained using a two-stage strategy, with initial rounds focusing on training the LSTM and CRF layers while keeping the pre-trained model parameters frozen, followed by joint fine-tuning of the entire model.
Datasets
The experiments were conducted on three datasets from the EvaHan2025 competition. These datasets are derived from historical records, the "Twenty-Four Histories," and classical Chinese medicine texts. The training data includes annotations for punctuation, word segmentation, and part-of-speech tagging. The paper notes that the availability of annotated corpora for ancient Chinese is significantly lower compared to modern languages, posing a challenge for model generalization.
Key Insights and Ideas
- GujiRoBERTa is a strong foundation: The paper highlights the effectiveness of using a pre-trained language model specifically designed for ancient Chinese, like GujiRoBERTa, as a starting point.
- LSTM and CRF are synergistic: The combination of LSTM for sequential modeling and CRF for structured prediction proves to be a powerful approach for ancient Chinese NER, outperforming baseline models.
- Class imbalance is a challenge: The authors acknowledge that the "Outside" tag (representing non-entity tokens) is significantly more prevalent than other entity tags, and they address this by using Focal Loss during training.
- Unidirectional LSTM's efficiency: The paper argues for the advantages of a unidirectional LSTM architecture, citing its superior inference speed, reduced training time, and better performance under low-resource conditions compared to bidirectional LSTMs and attention-based models.
Key Results
The proposed GujiRoBERTa-LSTM-CRF model achieved superior performance across multiple evaluation metrics on the EvaHan2025 datasets compared to the baseline SikuRoBERTa-BiLSTM-CRF model. For instance, on Dataset A, the proposed model achieved an F1-score of 83.17%, an improvement over the baseline's 81.48%. On Dataset B, the F1-score was 88.59% compared to 87.50%. While the paper notes a slight dip in precision on Dataset A compared to a bidirectional LSTM baseline, it attributes this to the inherent strengths of bidirectional architectures in global context modeling, while emphasizing the unidirectional model's superior computational efficiency and robustness.
Limitations and Future Work
The authors acknowledge several limitations:
- Limited Annotated Data: The scarcity of high-quality labeled datasets for ancient Chinese restricts the model's ability to generalize across different historical texts and domains.
- Domain-Specific Challenges: Ancient Chinese texts exhibit significant variations in style, terminology, and conventions across different periods and genres. A model trained on one specific dataset might not perform optimally on texts from other historical periods or literary traditions.
Future work could involve exploring more advanced data augmentation techniques to address the data scarcity issue and developing models that are more adaptable to the diverse linguistic characteristics of ancient Chinese texts. The paper also suggests further investigation into the advantages of unidirectional LSTM architectures for real-time applications.
Summary updated