TL;DR
This paper introduces HanTrans, a suite of pre-trained language models specifically designed for historical Chinese texts, addressing the scarcity of such resources. They demonstrate that these models significantly improve performance on downstream tasks like word segmentation and part-of-speech tagging for ancient Chinese, and importantly, show promising "zero-shot" cross-era transferability, meaning models trained on one era of Chinese can perform well on others.
What Problem Does This Paper Address?
Modern Natural Language Processing (NLP) heavily relies on large pre-trained language models (PLMs) like BERT. While these models excel at various tasks, their effectiveness is largely dependent on the vast amounts of training data available. For Chinese, most existing PLMs are trained on modern, simplified Chinese, with a significant underrepresentation of traditional Chinese, and a severe lack of historical Chinese texts (ancient, medieval, and early modern). This data imbalance creates a bottleneck for NLP applications dealing with historical Chinese documents, hindering research in fields like digital humanities, historical linguistics, and classical Chinese studies. The authors aim to bridge this gap by creating specialized PLMs for historical Chinese and investigating their transferability across different eras of the language.
Methods
The core of this work involves adapting the BERT architecture for historical Chinese. They achieve this through several key steps:
- Data Preparation: They curate and process four distinct corpora of Chinese texts spanning different historical eras: Ancient Chinese (pre-Qin to Western Han), Medieval Chinese (Eastern Han to Six Dynasties), Early Modern Chinese (Tang Dynasty onwards), and Modern Chinese.
- Vocabulary Expansion: To handle the unique vocabulary of historical texts, they extend the existing BERT vocabulary using a WordPiece tokenization strategy. This ensures that historical words not present in modern vocabularies are represented.
- Pre-training: They pre-train BERT-style models on these historical Chinese corpora using the Masked Language Modeling (MLM) objective. This standard MLM approach involves masking out tokens and training the model to predict them, thereby learning contextual representations.
- Downstream Task Fine-tuning: They fine-tune these pre-trained models on two specific downstream tasks: word segmentation and part-of-speech (POS) tagging. This involves adapting the pre-trained models to perform these specific linguistic analyses.
- Cross-Era Transferability Study: A significant aspect of their work is investigating how well models trained on one era of Chinese perform on texts from other eras without further training (zero-shot transfer). They also explore the benefits of training on a merged dataset encompassing all four eras.
- Visualization: They use dimensionality reduction techniques (PCA and t-SNE) to visualize the word embeddings of personal pronouns across different eras, offering insights into how semantic relationships evolve.
Datasets
The authors utilize four distinct Chinese corpora for their research:
- Ancient Chinese: Texts from pre-Qin to Western Han.
- Medieval Chinese: Texts from Eastern Han to the Six Dynasties.
- Early Modern Chinese: Texts from the Tang Dynasty onwards.
- Modern Chinese: A balanced corpus of modern Chinese.
These corpora are used for both pre-training the language models and fine-tuning them for downstream tasks. The paper also mentions the use of the Penn Chinese Treebank for POS tagging, though the primary focus is on their curated historical corpora.
Key Insights and Ideas
- Historical Chinese NLP is Feasible: The paper demonstrates that pre-trained language models can be successfully developed and applied to historical Chinese texts, opening up new avenues for research in this under-resourced area.
- Cross-Era Transferability is Promising: A key finding is that models trained on one era of Chinese can exhibit surprisingly good performance on texts from other eras, especially for tasks like word segmentation. This suggests a degree of shared linguistic structure across different historical periods.
- POS Tagging is More Era-Specific: Unlike word segmentation, POS tagging shows less robust cross-era transferability. This is attributed to significant differences in the tag sets and annotation schemes across historical periods, particularly the unique and extensive tag set for Ancient Chinese.
- Merged Datasets Offer Stability: Training on a combined dataset of all four eras results in models that are generally more stable and perform well across different test sets, even if they don't always achieve the absolute best performance on a specific era compared to models trained exclusively on that era's data.
- Pronoun Embeddings Reveal Semantic Stability: Visualizing pronoun embeddings shows that while there are subtle shifts, the relative positions of pronouns like "I," "you," and "he/she" remain remarkably consistent across different historical eras, suggesting core semantic relationships are preserved.
Key Results
- Improved Perplexity: The proposed HanTrans models significantly reduce perplexity scores on masked language modeling tasks for historical Chinese compared to baseline models trained only on modern Chinese. For instance, their
our/bert-base-han-chinesemodel trained on the Ancient Chinese set achieved a perplexity of 24.7588, a substantial improvement over the baseline's 233.6394. - Strong Downstream Performance: Fine-tuned HanTrans models achieve competitive F1 scores on word segmentation and POS tagging for historical Chinese. For example, their merged model (
our/bert-base-han-chinese-ws) achieved an F1 score of 97.4537 on the modern Chinese test set. - Zero-Shot Word Segmentation: The paper highlights successful zero-shot transfer for word segmentation. A model trained on Early Modern Chinese achieved an F1 score of 92.19 on Medieval Chinese, comparable to a model trained directly on Medieval Chinese (92.6538).
- Pronoun Visualization: PCA and t-SNE visualizations of pronoun embeddings show consistent relative positions across models trained on different eras, indicating preserved semantic relationships.
Limitations and Future Work
- Data Imbalance: The study acknowledges that the amount of data varies significantly across historical eras, with modern Chinese having the most and medieval Chinese the least. This imbalance can affect model performance, particularly for the less-resourced eras.
- POS Tagging Discrepancies: The significant differences in POS tag sets across eras limit cross-era transferability for this task. Future work could explore methods for harmonizing tag sets or developing more robust cross-lingual/cross-era POS tagging techniques.
- Exploration of Generative Models: The authors suggest exploring non-encoder-based architectures, such as text-generation models, for future research, potentially offering different perspectives and capabilities compared to BERT-style models.
- Broader Historical Coverage: While covering four eras, there's potential to expand to even finer-grained historical periods or other related Chinese dialects.
Summary updated