Introduction
Chinese Word Segmentation (CWS) is a fundamental prerequisite for downstream natural language processing tasks. While state-of-the-art segmenters achieve high accuracy on formal text, their performance degrades significantly on social media platforms like Weibo. This degradation is primarily driven by a high rate of out-of-vocabulary (OOV) words—including internet slang, "Chinglish" terms, and deliberate typos—alongside informal sentence structures. Traditional segmenters relying on static dictionaries struggle to keep pace with the rapid evolution of microblogging vocabulary.
Methodology
To address these challenges, we propose a cascaded Conditional Random Field (CRF) framework. This approach decomposes the segmentation task into two distinct stages: coarse-grained segmentation and refined OOV recognition.
Cascaded Architecture
- Coarse-Grained Segmentation (Model 1): A character-based CRF model is first trained on standard corpora. It identifies in-vocabulary words and establishes initial boundaries using a specialized 6-tag set.
- Refined OOV Recognition (Model 2): The second stage focuses on fragments left unsegmented or incorrectly segmented by the first pass. This model incorporates a "plug-in" user-edited dictionary, allowing for rapid adaptation to emerging trends without retraining the entire system.
Tagging Scheme
The system utilizes a 6-tag set to represent character positions within words. This granularity helps in disambiguating word boundaries more effectively than simpler tagging schemes.
Feature Engineering: Character Cohesion
A key innovation in this framework is the introduction of a Cohesion Ratio feature. This statistical measure quantifies the likelihood of adjacent characters forming a word based on their co-occurrence frequency in training data. By categorizing cohesion into Low, High, and Neutral levels, the CRF model can better "guess" the boundaries of OOV words based on character affinity, even when the specific word was not seen during training.
Experimental Setup
The models were trained using six months of the PKU "People's Daily" corpus (year 2000). For the user-edited dictionary, we utilized the "Sogou word bank," which contains over 278,000 entries including internet slang and celebrity names. Evaluation was performed on the SIGHAN 2012 Chinese MicroBlog segmentation task dataset.
Results and Discussion
The experimental results demonstrate the efficacy of both the cascaded approach and the cohesion features:
- Impact of Cohesion Features: Incorporating character cohesion features improved the F1 score on Weibo test data from 0.906 to 0.926.
- OOV Recovery: On standard PKU text, OOV recall increased from 0.847 to 0.860, confirming that cohesion is a strong signal for identifying novel words.
- Overall Performance: The final system, which integrated Weibo-specific samples into the training set, achieved a peak F1 score of 0.933.
Conclusion
The cascaded CRF framework provides a robust solution for the linguistic variability of social media text. By separating standard segmentation from OOV refinement and leveraging character cohesion, the system achieves high precision and recall on informal Chinese text. Future work will explore more computationally efficient ways to calculate character relationships and the use of active learning to strategically expand training data.
Summary updated