TL;DR
This paper tackles the challenge of normalizing variant characters in Classical Chinese texts, which often have multiple graphical forms for the same word. By leveraging parallel editions of texts and BERT embeddings, the authors demonstrate that context-aware models can learn to distinguish meaningful variant usage from arbitrary alterations, though such meaningful distinctions are surprisingly rare.
What Problem Does This Paper Address?
Classical Chinese, unlike modern languages, lacks a standardized orthography, leading to a high prevalence of "variant characters" (異體字). These are characters with different visual forms that represent the same morpheme or word. For tasks like searching, identifying parallel passages, or analyzing vocabulary, normalizing these variants into a single form is highly desirable. However, a naive approach of simply replacing all variants with a single canonical form risks losing crucial information. This is because some variants are indeed interchangeable, reflecting copyist preferences, while others are "quasi-variants" that are intentionally used to distinguish between different meanings or words. The core problem is to build an intelligent system that can differentiate these cases and normalize accordingly, without conflating meaningful distinctions.
Methods
The authors propose a two-stage approach:
- Dataset Construction: They build a dataset of potential "quasi-variant" character instances by aligning parallel editions of Classical Chinese texts. An algorithm identifies characters in one edition that correspond to multiple different characters in the other edition within a local context. This process filters out cases where variants are clearly interchangeable according to existing dictionaries (like the Dictionary of Chinese Character Variants - DCCV).
- Contextual Prediction: For the identified quasi-variant instances, they use contextualized BERT embeddings. A logistic regression model is trained on these embeddings to predict the correct variant usage. The core idea is to train the model on passages where a character is not differentiated (the "left-hand side" of a substitution profile) and test its ability to predict the differentiated variants (the "right-hand side") in other contexts. This tests whether the model can learn meaningful conventions from context.
Datasets
The primary dataset is constructed from parallel editions of Classical Chinese texts sourced from the Wenyuange copy of the Siku quanshu (SKQS) and the 1919 edition of the Sibu congkan (SBCK), available through the Kanseki repository. This corpus contains 286 works with aligned editions. The authors do not introduce a new benchmark dataset but rather create a dataset of substitution profiles and their occurrences from existing parallel texts.
Key Insights and Ideas
- Meaningful Differentiation is Rare: Contrary to initial expectations, the study reveals that truly meaningful semantic distinctions between variant characters are surprisingly uncommon in the surveyed corpus. Most observed variations appear to be arbitrary or due to transcription errors.
- Context is Key, But Not Always Sufficient: BERT embeddings, when combined with logistic regression, can learn to predict differentiated variants in specific contexts, indicating that some conventions are discernible. However, the ability to learn these conventions is often document-specific.
- Idiosyncrasies Dominate: The failure of learned conventions to transfer well between documents suggests that copyists and editors had significant freedom in choosing variants, and idiosyncratic preferences are more prevalent than universal orthographic norms.
- Dictionary Limitations: The study highlights cases where a distinction is revealed by the model that is not captured by existing dictionaries, underscoring the limitations of relying solely on lexicographical resources for normalization.
Key Results
- The first experiment, testing single-document learning, found that for a significant number of document-profile combinations, a logistic regression model could predict differentiated variants with accuracy significantly better than a naive classifier (77 combinations for the final hidden layer output).
- However, the second experiment, testing cross-document transferability, showed that learned conventions rarely generalize. Only a few profiles (notably those involving "li" 歷/曆) demonstrated consistent cross-document transferability.
- The accuracy for successful single-document predictions ranged from 51.2% to 100% (mean 89.2%).
- The study identified 108 substitution profiles that showed potential for meaningful differentiation after filtering out clearly interchangeable variants.
Limitations and Future Work
- Digitalization Artifacts: The authors acknowledge that without comparing original manuscripts to digital editions, it's difficult to ascertain whether some visually similar variants are due to inconsistencies in the digitalization process.
- Recall of Extraction: The lack of a manually annotated dataset means the recall of their quasi-variant extraction algorithm is unknown; low-frequency variants might be missed.
- Model Interpretability: It's unclear what specific linguistic features the BERT model uses to make its predictions, and whether these are truly meaningful to human scholars.
- Future Work: The authors suggest rerunning experiments with the full list of substitution profiles (including those deemed interchangeable by the DCCV) to better understand the prevalence of arbitrary vs. meaningful variation. They also propose exploring methods to improve cross-document transferability, potentially by training on multiple documents. Further research is needed to determine how to establish "specialized forms" for normalization when conventions are not uniform.
Summary updated