TL;DR
This paper tackles the problem of "meaningless words" generated by Chinese word segmentation systems, a common issue when models encounter unseen words. They propose a novel word lattice-based framework that incorporates "wordhood" features derived from lexicons and large corpora to significantly reduce these errors, achieving state-of-the-art performance and a notable reduction in meaningless word generation.
What Problem Does This Paper Address?
Chinese, unlike many Western languages, does not use explicit spaces to delineate words. This makes word segmentation (identifying word boundaries) a crucial first step for many Natural Language Processing (NLP) tasks. While statistical models are good at generalizing to new words, they often err by splitting existing words into nonsensical sub-units or merging unrelated characters into a single meaningless "word." This phenomenon, termed "meaningless word generation," degrades the quality of downstream NLP applications. The paper aims to address this by developing a more robust Chinese word segmentation and part-of-speech (POS) tagging system that specifically targets and reduces these erroneous, meaningless word creations.
Methods
The core of the proposed system is a word lattice-based framework. This framework combines a character-based model with a pruned word-based model.
- Character-Based Model: This model generates an initial set of possible segmentations and POS tags for each character, represented as a word lattice. This lattice contains multiple potential word sequences, allowing for more flexibility than a single output.
- Word Lattice Generation: A novel method is introduced to generate this lattice, controlled by a parameter
d. This parameter balances the lattice scale (number of possible paths) with the quality of the paths, aiming to ensure that correct segmentations are likely to be included. - Pruned Word-Based Model: This model then searches within the generated word lattice to find the optimal segmentation and POS tagging sequence.
- Wordhood Features: The key innovation lies in the features used by the word-based model. These features are designed to assess the "wordhood" or likelihood of a string being a valid word. They are derived from:
- General Lexicon (SogouW): A large lexicon providing word categories.
- Wikipedia: Used as a lexicon for named entities and domain-specific terms.
- Large-scale Raw Corpus (SogouT): A massive corpus (200 billion characters) used to calculate word frequencies and contextual patterns (Restricted Accessor Variety - RAV).
These wordhood features help the model distinguish between legitimate words and spurious, meaningless segmentations.
Datasets
- Penn Chinese Treebank 5.0 (CTB 5.0): Used for training, development, and testing. This is a standard benchmark dataset for Chinese NLP tasks.
- SogouW: A large, publicly available Chinese lexicon used for wordhood features.
- Chinese Wikipedia: Used as a lexicon for wordhood features.
- SogouT: A large-scale raw corpus of web pages (over 200 billion characters) used for wordhood features.
Key Insights and Ideas
- Meaningless words are a serious problem: The paper highlights that errors creating meaningless words are more detrimental than errors that produce grammatically correct but potentially suboptimal segmentations.
- Wordhood is key to reducing meaningless words: By explicitly modeling the likelihood of a string being a "word" using external resources, the system can better avoid generating nonsensical units.
- Word lattices offer a flexible search space: Using a word lattice allows the system to explore multiple segmentation possibilities, making it more robust than single-pass methods.
- Combining diverse resources is effective: Leveraging general lexicons, Wikipedia, and large raw corpora provides complementary information for assessing wordhood.
- Feature engineering matters: The careful design of wordhood features, particularly those derived from contextual patterns in large corpora (RAV), significantly improves performance.
Key Results
- Significant Reduction in Meaningless Words: The proposed system achieved a 62.9% reduction in meaningless word generation compared to a baseline character-based model.
- Improved Segmentation Performance: The F1 measure for segmentation was increased to 0.984.
- State-of-the-Art Performance: The system outperformed previous work on the Penn Chinese Treebank 5.0 in closed-test settings for both segmentation (Seg) and S&T.
- Efficiency: The system achieved a speed of 107.77 sentences per second on the test set, demonstrating its efficiency despite the complex framework.
- Feature Effectiveness: Experiments showed that features from general lexicons and raw corpora were helpful, while Wikipedia features were less impactful for the specific task.
Limitations and Future Work
- Wikipedia Feature Impact: The paper notes that Wikipedia features were not as helpful as expected, suggesting further investigation into how to best utilize this resource.
- Computational Cost: While efficient, the word lattice generation and pruned word-based model still involve significant computation. The authors mention that a large
d(lattice scale parameter) increases decoding time. - Domain Adaptation: The current experiments are on the CTB corpus, which primarily consists of news and magazines. The authors suggest that for documents like blogs, microblogs, or domain-specific texts, the challenge of meaningless words might be even greater, and they plan to test their method on such documents in the future.
- Feature Combination: The paper explored combinations of wordhood features, indicating that further tuning of these combinations could yield even better results.
Summary updated