Introduction
Chinese Word Segmentation (CWS) is the foundational task in Chinese Natural Language Processing. Unlike English, Chinese text lacks explicit word boundaries, requiring computational models to determine where words begin and end. A significant challenge in this process is Covering Ambiguity, which occurs when a character string can be interpreted either as a single word ($w$) or as a sequence of smaller constituent words ($w_1...w_n$). For example, the string "中将" (zhōngjiàng) can mean "lieutenant general" (one word) or "middle" and "will/about to" (two words).
This paper presents a high-accuracy method for resolving such ambiguities by reframing the problem as a Word Sense Disambiguation (WSD) task. By leveraging contextual information through a Vector Space Model (VSM) and semantic smoothing, the proposed approach achieves state-of-the-art performance.
Methodology
The core of the proposed system is a Vector Space Model designed to represent the context surrounding an ambiguous string. The methodology consists of four primary components:
1. Contextual Windowing and Regional Division
To capture the syntactic and semantic influence of neighboring words, the model defines a window of $\pm 3$ words around the ambiguous term. This window is divided into four distinct regions ($R_1$ to $R_4$), allowing the model to account for the positional importance of specific context words.
2. Modified TF-IDF Weighting
Standard TF-IDF weighting was found insufficient for this task. The authors introduced two modifications:
- Logarithmic TF: Applied to prevent high-frequency words from disproportionately influencing the vector.
- Exponential IDF Variation: A novel formula based on the mean and standard deviation of a word's appearance across different segmentation forms, helping to identify truly discriminative context features.
3. Semantic Smoothing via Thesaurus
To address the "data sparseness" problem—where many relevant context words appear too infrequently for statistical significance—the model utilizes the TongYiCiCiLin Chinese thesaurus. Low-frequency words are replaced with high-level semantic codes, effectively grouping synonyms to create a denser feature space.
4. Hybrid Strategy
The system employs a dual-model approach based on an imbalance ratio ($u$). If one segmentation form is overwhelmingly dominant in the training data, a "default" model is used; otherwise, the sophisticated "core" VSM model is applied to resolve the ambiguity.
Experimental Results
The model was evaluated on 90 frequent cases of covering ambiguity using a dataset of approximately 97,000 manually labeled sentences. The proposed approach demonstrated significant improvements over traditional methods.
As shown in the results, the proposed model achieved an overall accuracy of 96.58%. This drastically outperforms the standard Hidden Markov Model (HMM), which achieved only 68.48% on the same test set. The failure of the HMM highlights that local statistical transitions are insufficient for resolving covering ambiguities, whereas the VSM's broader contextual and semantic features are highly effective.
Conclusion
By treating Chinese word segmentation as a disambiguation task, this research demonstrates that deep contextual analysis and semantic smoothing can overcome the limitations of traditional statistical models. The high accuracy and computational efficiency of the VSM approach make it a viable module for integration into full-scale CWS systems.
Summary updated