TL;DR
This paper introduces a Unicode-based Chinese word segmentor designed to handle Simplified, Traditional, and mixed-mode Chinese text with high accuracy and adaptability. Its modular design, combined with a novel new word extractor and a unified dictionary approach, allows for efficient training and adaptation to new domains with minimal manual effort.
What Problem Does This Paper Address?
Chinese word segmentation is notoriously difficult due to inherent ambiguities, especially when dealing with names, foreign terms, and numerical expressions. Traditional approaches often require extensive manual effort to train systems for specific language environments or to handle variations like Simplified and Traditional Chinese. This paper addresses the need for a segmentor that is not only accurate but also flexible, adaptive to different domains, and capable of processing mixed-mode Chinese text without maintaining separate dictionaries for each script.
Methods
The core of the system is a modular design, comprising a segmentor and a data manager. The segmentor itself has three main components:
- Pre-processor: Handles specialized tasks like recognizing personal names, numbers, and time expressions using a divide-and-conquer strategy.
- Kernel: Implements various segmentation algorithms, including maximum and minimum matching in forward and backward modes, and can incorporate part-of-speech tagging for disambiguation.
- Post-processor: Applies morphological rules that are difficult to handle with dictionary lookups alone.
A key innovation is the new word extractor, which uses statistical data (bi-grams, tri-grams, quadra-grams) to semi-automatically identify and extract new words and names from running text. This significantly reduces the need for manual lexicon updates. The system also employs a unified dictionary approach, storing words in their "canonical form" and using conversion tables to handle variants between Simplified and Traditional Chinese, thus avoiding the need for separate dictionaries.
Datasets
The paper mentions using a basic dictionary and tagging data provided by Peking University. For evaluation, the system was tested on four corpora: AS, CTB (UPenn), HK, and PK. The authors also mention using one year of Hong Kong Commercial Daily News data for training the new word extractor.
Key Insights and Ideas
- Unified Dictionary for Script Diversity: The decision to use a single dictionary with canonical forms and conversion tables is a clever way to handle both Simplified and Traditional Chinese, as well as mixed-mode text, without the overhead of managing multiple lexicons.
- Adaptive New Word Extraction: The built-in new word extractor is crucial for adaptability. By automatically identifying new terms, it allows the segmentor to quickly adjust to new domains or evolving language usage without extensive manual retraining.
- Modular Design for Flexibility: The modular architecture allows for independent development and fine-tuning of different components, making the system easier to extend and maintain. New rules or modules can be added with minimal impact on the rest of the system.
- Pre-processing for Special Cases: Explicitly handling personal names, numbers, and time expressions in the pre-processing stage is a practical approach to improve accuracy on these challenging categories.
Key Results
The system demonstrated strong performance across the four tested corpora, achieving F-scores ranging from 0.872 to 0.925. The worst performance was observed on the CTB data, attributed to word granularity differences. The best performance was on the PK data, likely due to a more similar dictionary. The system's segmentation speed averaged around 7,500 characters per second for three of the corpora, with the PK data showing a slightly higher rate.
Limitations and Future Work
- CTB Data Granularity: The paper acknowledges that differences in word granularity in the CTB dataset posed a challenge.
- Out-of-Vocabulary (OOV) Issues: For the AS data, out-of-vocabulary mistakes were a significant problem, suggesting that further improvements in new word extraction or handling unseen words are needed.
- Training Data Dependency: While the system is adaptive, its performance is still influenced by the similarity of the training data to the target domain, as seen with the PK and AS results.
The authors suggest that the system's modular design and the inclusion of a new word extractor make it easily adaptable to new environments, implying that further domain-specific tuning and expansion of the new word extractor's capabilities would be natural directions for future work.
Summary updated