Chinese Segmentation and New Word Detection using Conditional Random Fields.

Fuchun Peng, Fangfang Feng, Andrew McCallumCOLING 2004 · OpenReview

Abstract

Chinese Word Segmentation (CWS) is a fundamental task in Chinese natural language processing. This paper presents a principled framework based on Conditional Random Fields (CRFs) that effectively integrates dictionary-based domain knowledge with statistical learning. By modeling the conditional probability of entire label sequences, the approach avoids the limitations of local classification. Furthermore, the authors introduce an integrated probabilistic method for detecting out-of-vocabulary (OOV) words, achieving state-of-the-art results across multiple benchmark datasets from the 2003 SIGHAN Bake-off.

Introduction

Unlike many Western languages, Chinese text is written as a continuous string of characters without explicit word boundaries. Consequently, CWS is the essential first step for downstream tasks such as machine translation and information retrieval. Historically, CWS methods have been divided into heuristic dictionary-based approaches—which are fast but struggle with unknown words—and statistical machine learning models—which are robust but often fail to incorporate linguistic constraints or model global dependencies effectively.

This research bridges these two paradigms using Linear-Chain CRFs. CRFs provide a flexible framework to incorporate arbitrary features, such as external lexicons, while maintaining the mathematical rigor of sequence modeling.

Methodology

The core of the proposed system is a sequence tagging model where each character is labeled as either START (beginning of a word) or NONSTART (continuation of a word).

Feature Engineering

The model leverages two types of features:

  1. Closed Features: Derived strictly from the training data, including character unigrams and bigrams.
  2. Open Features: Derived from external resources, such as lists of names, places, and parts of speech (POS).

To capture complex contextual patterns, the authors utilize feature conjunctions, combining the current character with its immediate neighbors (e.g., the current and next character in the lexicon).

Probabilistic New Word Detection (NWD)

A significant contribution of this work is the integrated approach to identifying out-of-vocabulary words. Instead of treating NWD as a separate post-processing step, the system calculates a confidence score for segmented words using a constrained forward-backward algorithm. High-confidence sequences not found in the dictionary are flagged as "new words," added to a temporary lexicon, and the segmentation is re-run to refine the final output. This two-pass approach ensures that identifying unknown words directly improves the accuracy of the surrounding known words.

Experimental Setup

The system was evaluated on four diverse datasets from the 2003 SIGHAN Chinese Word Segmentation Bake-off, representing different dialects and encoding standards.

Summary of the SIGHAN 2003 benchmark datasets used for evaluation.

As shown in the table above, the datasets vary significantly in size and OOV rates, providing a rigorous test for the model's robustness and generalization capabilities.

Results and Discussion

The CRF-based approach demonstrated superior performance across all benchmarks. The global optimization inherent in CRFs prevents "illegal" tag transitions, leading to more coherent segmentations than local Maximum Entropy models.

Performance metrics (Precision, Recall, F1, and OOV Recall) for the CRF model on Closed and Open tracks.

Key findings include:

  • State-of-the-Art Accuracy: The system achieved the highest F1-scores in several competition tracks, notably in the AS open and HK closed categories.
  • Impact of NWD: The integrated new word detection significantly improved results. For the CTB dataset, the F1-score increased from 0.792 to 0.849 upon enabling NWD.
  • Robustness: The model maintained high performance even in cross-dataset scenarios, proving more resilient than traditional generative models.

Conclusion

Conditional Random Fields offer a powerful and unified framework for Chinese word segmentation. By treating dictionaries as features rather than rigid rules and integrating new word detection directly into the probabilistic model, this approach achieves high accuracy and robustness. While training complexity remains a challenge, the ability to incorporate diverse linguistic knowledge makes CRFs a state-of-the-art choice for CWS. Future work will explore more efficient training algorithms and the integration of specialized named entity recognition.

Summary updated

Abstract

Fuchun Peng, Fangfang Feng, Andrew McCallum. COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics. 2004.
BibTeX
@inproceedings{peng2004chinese,
  author = {Fuchun Peng and Fangfang Feng and Andrew McCallum},
  title = {{Chinese Segmentation and New Word Detection using Conditional Random Fields.}},
  booktitle = {COLING},
  year = {2004},
  url = {https://openreview.net/forum?id=rlpW8bdOth},
}
Figure