Two-Character Chinese Word Extraction Based on Hybrid of Internal and Contextual Measures

Shengfen Luo, Maosong SunSIGHAN 2003 · OpenReview

TL;DR

This paper tackles the challenge of automatically extracting two-character words from Chinese text, a crucial step for natural language processing. They propose a novel hybrid approach that combines statistical "internal" measures (analyzing word structure) with "contextual" measures (analyzing word usage in its surroundings), achieving a significant F-measure of 68.48%.

What Problem Does This Paper Address?

Chinese, unlike many Western languages, doesn't use spaces to delineate words. This makes automatic word segmentation a fundamental and challenging task for natural language processing (NLP). As Chinese society rapidly evolves, new words are constantly being coined, meaning static lexicons quickly become outdated. Therefore, developing robust methods for automatically extracting these new words from large text corpora is essential for keeping NLP systems up-to-date and effective. This paper specifically focuses on two-character words, which represent a significant portion of the Chinese lexicon and a common word-formation pattern.

Methods

The authors explore two main categories of statistical measures for word extraction:

  • Internal Measures: These assess the "associative strength" within a potential word (e.g., how likely two characters are to appear together). They tested nine widely adopted measures, including Frequency, Mutual Information (MI), and Dice Formula.
  • Contextual Measures: These evaluate how a word fits into its surrounding text, specifically using "left/right entropy" to measure the diversity of characters appearing before and after a potential word.

The core of their contribution lies in combining these measures:

  1. Direct Combination: Initially, they tried a weighted sum of the nine internal measures.
  2. Interval-based Combination: A more sophisticated approach where each measure's value range is discretized into intervals. Each interval is assigned a "probability" of representing a word, and these probabilities are then combined using weights.
  3. Hybrid Approach: The most advanced method integrates the best interval-based combination of internal measures with the contextual measure (left/right entropy).

Crucially, they employ a genetic algorithm to automatically optimize the weights for combining measures and to tune thresholds for contextual measures, avoiding manual parameter tuning.

Datasets

  • PDR9596: A large raw corpus of People Daily from 1995-1996 (approx. 50.0M characters) used for training Chinese character bigram matrices.
  • PDA98J: A manually word-segmented corpus of People Daily from January 1998 (approx. 1.3M characters) used for evaluating word extraction performance and as a gold standard.
  • TS1: A training set derived from PDR9596, containing 196,977 distinct bigrams, with 17,333 identified as two-character words in PDA98J.
  • TS2: An independent test set used for open testing of the final hybrid model.

Key Insights and Ideas

  • Mutual Information is a strong single predictor: Among the nine internal measures, Mutual Information (MI) consistently performed the best on its own.
  • Combination is key: Simply using the best single measure (MI) is insufficient. Combining multiple internal measures, especially using an interval-based approach, significantly boosts performance.
  • Context matters: Integrating left/right entropy (a contextual measure) with internal measures provides a substantial improvement, indicating that how a word is used in context is as important as its internal structure.
  • Automatic optimization is powerful: Using a genetic algorithm to tune weights and thresholds automates a complex and tedious process, leading to better and more robust results.
  • Entropy-based discretization is superior: Variable-length intervals derived from entropy-based discretization performed much better than equal-length intervals for combining measures.

Key Results

  • The best single internal measure, Mutual Information (MI), achieved an F-measure of 54.77% on TS1.
  • The best direct combination of internal measures yielded an F-measure of 55.44%.
  • The interval-based combination of seven internal measures (excluding SA and SCP) achieved an F-measure of 57.77% on TS1.
  • The final hybrid model, combining the interval-based internal measures with left/right entropy, achieved an F-measure of 68.48% on TS1.
  • In an open test on TS2, the hybrid model achieved an F-measure of 68.48%, a significant improvement over MI (57.82%) and the best internal combination (59.87%).

Limitations and Future Work

  • Focus on two-character words: The current work is limited to extracting only two-character words.
  • Contextual measure reliability: The authors note that left/right entropy can be less reliable when context occurrences are insufficient.
  • Future Work: The authors plan to extend their method to automatically extract multi-character Chinese words. They also suggest incorporating other resources like lexicons and semantic information to further enhance performance.

Summary updated

BibTeX
@inproceedings{luo2003two,
  author = {Shengfen Luo and Maosong Sun},
  title = {{Two-Character Chinese Word Extraction Based on Hybrid of Internal and Contextual Measures}},
  booktitle = {SIGHAN},
  year = {2003},
  url = {https://openreview.net/forum?id=wWJcxdjoIu},
}
Figure