Abstract
Chinese Word Segmentation (CWS) is a fundamental task in Chinese natural language processing. This paper presents a principled framework based on Conditional Random Fields (CRFs) that effectively integrates dictionary-based domain knowledge with statistical learning. By modeling the conditional probability of entire label sequences, the approach avoids the limitations of local classification. Furthermore, the authors introduce an integrated probabilistic method for detecting out-of-vocabulary (OOV) words, achieving state-of-the-art results across multiple benchmark datasets from the 2003 SIGHAN Bake-off.
Introduction
Unlike many Western languages, Chinese text is written as a continuous string of characters without explicit word boundaries. Consequently, CWS is the essential first step for downstream tasks such as machine translation and information retrieval. Historically, CWS methods have been divided into heuristic dictionary-based approaches—which are fast but struggle with unknown words—and statistical machine learning models—which are robust but often fail to incorporate linguistic constraints or model global dependencies effectively.
This research bridges these two paradigms using Linear-Chain CRFs. CRFs provide a flexible framework to incorporate arbitrary features, such as external lexicons, while maintaining the mathematical rigor of sequence modeling.
Methodology
The core of the proposed system is a sequence tagging model where each character is labeled as either START (beginning of a word) or NONSTART (continuation of a word).
Feature Engineering
The model leverages two types of features:
- Closed Features: Derived strictly from the training data, including character unigrams and bigrams.
- Open Features: Derived from external resources, such as lists of names, places, and parts of speech (POS).
To capture complex contextual patterns, the authors utilize feature conjunctions, combining the current character with its immediate neighbors (e.g., the current and next character in the lexicon).
Probabilistic New Word Detection (NWD)
A significant contribution of this work is the integrated approach to identifying out-of-vocabulary words. Instead of treating NWD as a separate post-processing step, the system calculates a confidence score for segmented words using a constrained forward-backward algorithm. High-confidence sequences not found in the dictionary are flagged as "new words," added to a temporary lexicon, and the segmentation is re-run to refine the final output. This two-pass approach ensures that identifying unknown words directly improves the accuracy of the surrounding known words.
Experimental Setup
The system was evaluated on four diverse datasets from the 2003 SIGHAN Chinese Word Segmentation Bake-off, representing different dialects and encoding standards.
As shown in the table above, the datasets vary significantly in size and OOV rates, providing a rigorous test for the model's robustness and generalization capabilities.
Results and Discussion
The CRF-based approach demonstrated superior performance across all benchmarks. The global optimization inherent in CRFs prevents "illegal" tag transitions, leading to more coherent segmentations than local Maximum Entropy models.
Key findings include:
- State-of-the-Art Accuracy: The system achieved the highest F1-scores in several competition tracks, notably in the AS open and HK closed categories.
- Impact of NWD: The integrated new word detection significantly improved results. For the CTB dataset, the F1-score increased from 0.792 to 0.849 upon enabling NWD.
- Robustness: The model maintained high performance even in cross-dataset scenarios, proving more resilient than traditional generative models.
Conclusion
Conditional Random Fields offer a powerful and unified framework for Chinese word segmentation. By treating dictionaries as features rather than rigid rules and integrating new word detection directly into the probabilistic model, this approach achieves high accuracy and robustness. While training complexity remains a challenge, the ability to incorporate diverse linguistic knowledge makes CRFs a state-of-the-art choice for CWS. Future work will explore more efficient training algorithms and the integration of specialized named entity recognition.
Summary updated