Chinese Word Segmentation without Using Lexicon and Hand-crafted Training Data

Maosong Sun, Dayang Shen, Benjamin K. TsouCOLING 1998

TL;DR

This paper introduces a novel Chinese word segmentation algorithm that completely eschews lexicons and hand-crafted linguistic resources. By leveraging statistical measures like mutual information and a new "difference of t-score" metric derived solely from raw Chinese corpora, the algorithm achieves acceptable accuracy, offering a promising avenue for improving robustness and adaptability in Chinese NLP systems.

What Problem Does This Paper Address?

Chinese word segmentation is a fundamental preprocessing step for virtually all Chinese Natural Language Processing (NLP) tasks. Unlike English, where words are separated by spaces, Chinese characters are concatenated, making it challenging to identify word boundaries. Traditional approaches heavily rely on large, manually curated lexicons and hand-annotated corpora. While effective, these resources are time-consuming and expensive to create and maintain, and they often struggle with data incompleteness, sparseness, and domain adaptation issues. This paper aims to address the significant human effort and resource limitations by developing a fully automatic segmentation method that learns from raw text alone.

Methods

The core of the proposed algorithm lies in two statistical measures:

  • Mutual Information (MI): This classic measure quantifies the statistical dependency between two characters. A higher MI between adjacent characters suggests a stronger likelihood they form a word. The paper uses the standard formula: mi(x: y) = log2(p(x, y) / (p(x)p(y))), where p denotes probabilities derived from character co-occurrences in the corpus.
  • Difference of t-score (dts): Recognizing the limitations of MI, especially in "intermediate ranges" where associations are ambiguous, the authors introduce a novel metric. The t-score (tsxz(y)) measures the binding tendency of character y in the context of x and z. The dts(x:y) is then defined as the difference between the t-score of x in its context (vxy) and the t-score of y in its context (xyw). This metric is context-dependent and aims to capture the "competition" between characters for forming words.

The algorithm integrates these measures in a two-round process. The first round uses a complex set of rules based on the combined values of MI and dts, along with concepts of "local maximum" and "local minimum" of dts, to tentatively mark character pairs as "bound" (part of a word) or "separated" (word boundary). The second round handles unmarked locations, primarily using MI, and refines decisions based on adjacent markings and context. The constants and thresholds used in the rules are determined empirically.

Datasets

The paper utilizes a large, unspecified news corpus (approximately 20 million Chinese characters) for deriving the statistical data (character bigram probabilities) needed to calculate MI and dts. For evaluation, they use 100 Chinese sentences, totaling 1588 characters (1587 potential locations between character pairs), which are randomly selected and kept separate from the training corpus.

Key Insights and Ideas

  • MI's limitations: The paper clearly demonstrates that mutual information alone is insufficient for accurate segmentation, especially in ambiguous cases. This motivates the development of more sophisticated, context-aware measures.
  • The power of context-dependent measures: The introduction of the "difference of t-score" (dts) highlights the importance of local context in word formation. By considering the surrounding characters, dts offers a more nuanced view of character associations.
  • Local extrema for segmentation: The concepts of "local maximum" and "local minimum" of dts are particularly insightful. They suggest that the "peak" or "valley" of a character's binding tendency within its local context can be a strong indicator of whether it's part of a word or a boundary.
  • Fully automatic approach: The most significant contribution is the demonstration that high-quality segmentation can be achieved without any human-engineered resources. This has major implications for reducing development costs and improving adaptability to new domains or evolving language.

Key Results

The algorithm achieves an accuracy of 91.75% on the test set of 1587 character locations. This is considered "acceptable" by the authors, especially given the extreme condition of using no external linguistic resources. The paper also presents distributions of dts types across different MI/dts regions, offering insights into how the algorithm categorizes character pairs.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Accuracy: While acceptable, 91.75% accuracy is still lower than state-of-the-art systems that utilize lexicons and hand-crafted data.
  • Parameter tuning: The algorithm relies on empirically determined constants and thresholds, which might require further refinement.
  • Integration: The authors express a strong desire to integrate their approach as a module into existing Chinese segmenters to improve their performance, particularly in handling unknown words and adapting to new domains.
  • Further study: Future work includes enlarging the training corpus, refining the algorithm by studying the relationship between MI and dts more deeply, and exploring the "blind area" of MI more effectively.

Summary updated

BibTeX
@inproceedings{sun1998chinese,
  author = {Maosong Sun and Dayang Shen and Benjamin K. Tsou},
  title = {{Chinese Word Segmentation without Using Lexicon and Hand-crafted Training Data}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1998},
}
Figure