Abstract
Chinese Word Segmentation (CWS) is a fundamental task in natural language processing, yet its performance often degrades when applied to literary texts due to the high frequency of novel-specific Out-of-Vocabulary (OOV) words. This paper presents a two-stage framework that utilizes a "double-propagation" algorithm to iteratively mine character names and location-specific patterns from raw text. By integrating these as "plug-in" features, the proposed method achieves a 30.2% error reduction in OOV recall and operates 36x faster than traditional domain adaptation methods.
Introduction
Unlike English, Chinese text lacks explicit word delimiters, making CWS the essential first step for downstream tasks. While state-of-the-art segmentors perform exceptionally well on news corpora, they struggle with novels where OOV words—such as unique character names, fictional locations, and invented organizations—are prevalent. Traditional "self-training" methods, which involve retraining models on their own predictions, are computationally prohibitive for large-scale web novel processing. This research addresses the need for a fast, effective adaptation method that handles the unique vocabulary of literature without requiring extensive manual annotation.
Methodology
The authors propose a two-stage framework that treats novel-specific entities as dynamic features rather than requiring a full model overhaul.
- Baseline Model: A joint segmentor and Part-of-Speech (POS) tagger is initially trained on standard news data.
- Double-Propagation Mining: This iterative bootstrapping process is the core innovation. It begins by using known context patterns (e.g., "went to [x]'s side") to identify potential new noun entities. These newly discovered entities are then used to identify further context patterns. The cycle repeats until convergence.
- Enhanced Segmentor: Instead of a full model overhaul, the mined words and patterns are incorporated as additional feature templates. This allows the model to recognize novel-specific names based on local context.
Key Insights: The Power of Context
A primary observation of this work is that while a specific name (e.g., "Tianbuyi") may be unknown to a model trained on news, the linguistic patterns surrounding it are often familiar and predictable.
By leveraging these patterns, the system avoids "catastrophic forgetting"—a common issue in domain adaptation where a model loses its general knowledge while learning new domain-specific information. The "plug-in" feature approach ensures the model remains robust across different texts.
Datasets and Evaluation
The system was trained on the People’s Daily Corpus (January 1998) and evaluated against five popular contemporary Chinese novels: Zhuxian (ZX), Fanren Xiuxian Zhuan (FR), XunLong Ji (XL), Douluo Dalu (DL), and Juedai Shuangjiao (JD). Gold-standard test sets were created by manually annotating one chapter from the middle of each novel.
Results
The proposed method significantly outperformed both the baseline and standard self-training techniques:
- OOV Recall: Achieved a 30.2% relative error reduction, demonstrating a superior ability to identify new names and places.
- Accuracy: Segmentation and POS tagging errors were reduced by 18.4% and 14.6%, respectively.
- Efficiency: While traditional retraining took approximately 6 hours per novel, this feature-based approach required only 10 minutes—a 36x speedup.
- Precision: Noun mining achieved approximately 95% precision for frequent entities.
Limitations and Future Work
Despite its success, the system occasionally misidentifies character names that resemble common nouns, such as "Xiaoyuer" (Little Fish). Future research will explore broader context patterns and the integration of external linguistic knowledge, such as surname lists, to further improve entity recognition precision.
Summary updated