Unigram Language Model for Chinese Word Segmentation

Aitao Chen, Yiping Zhou, Anne Zhang, Gordon SunSighan@ijcnlp 2005

TL;DR

This paper presents a Chinese word segmentation system that leverages a unigram language model to resolve ambiguities. The system is enhanced with pre-processing modules for named entity and numeric expression recognition, and post-processing steps to refine segmentation, achieving strong results in a major Chinese word segmentation bakeoff.

What Problem Does This Paper Address?

Chinese, unlike many Western languages, does not use spaces to delineate words. This means that a sequence of characters can be segmented into words in multiple ways, leading to ambiguity. For example, "北京大学" can be segmented as "北京大学" (Peking University) or "北京/大/学" (Beijing/big/learn). Accurately segmenting text is a crucial first step for many downstream Natural Language Processing (NLP) tasks in Chinese. This paper tackles the challenge of building a robust and accurate Chinese word segmentation system, particularly focusing on improving performance through sophisticated pre- and post-processing techniques.

Methods

The core of the system is a unigram language model. This model assumes that words in a sentence occur independently and estimates the probability of a segmentation by multiplying the probabilities of individual words. The system chooses the segmentation with the highest overall probability.

To enhance this core model, the authors incorporated several key components:

  • Pre-processors:
    • Named Entity Recognizers (NER): These modules identify and extract names of people, places, and organizations. They are built using the maximum entropy model.
  • Numeric Expression Recognizer: This module extracts numeric expressions and is implemented as a finite state automaton.
  • These extracted entities are temporarily added to the segmentation dictionary to guide the segmentation process.
  • Post-processors:
    • Single Character Combination: Sequences of single characters that frequently appear within words (but rarely as standalone words) are combined into a single unit.
  • Consistency Checking: If the system's segmentation of a text fragment differs from the segmentation in the training data, the training data's segmentation is used. This leverages the annotated training data as a form of "ground truth" for known segments.

Datasets

The paper mentions participation in several tasks using different corpora:

  • Academia Sinica corpus (AS): Used for both closed and open tasks.
  • City University of Hong Kong corpus (CITYU): Used for closed and open tasks.
  • Peking University Corpus (PKU): Used for closed and open tasks.
  • Microsoft Research corpus (MSR): Used for closed and open tasks.

The training data for these corpora was used to build dictionaries, word frequency counts, and train the various models. For the open tasks, user dictionaries were also employed.

Key Insights and Ideas

  • Leveraging External Knowledge: The success of the system hinges on effectively integrating external knowledge through pre- and post-processing. Simply relying on a unigram model with a base dictionary is insufficient.
  • The Power of Named Entities: Explicitly recognizing and incorporating named entities (people, places, organizations) and numeric expressions significantly improves segmentation accuracy, as these are often complex or multi-character words.
  • Consistency as a Refinement Tool: Using the training data's segmentation as a fallback for ambiguous segments (consistency checking) proved to be a powerful post-processing step, especially for closed tasks where the training data is directly relevant.
  • User Dictionaries for Open Tasks: For open tasks, where new words can appear, a user dictionary significantly boosts performance, demonstrating the value of incorporating domain-specific or previously unseen vocabulary.
  • Trade-offs in Post-processing: While combining single characters can recover some words, it can also lead to false positives, especially if those recovered words are already handled by other mechanisms like user dictionaries in open tasks.

Key Results

The paper reports strong performance in the second international Chinese word segmentation bakeoff. For example, the msr-open-b run achieved an F-score of 0.966, and the pku-open-a run achieved 0.956. The tables in the paper detail the performance across various runs and tasks, showing consistent improvements as more features (like numeric expression recognition, person/place name recognition, and consistency checking) are added. The authors highlight that the PKU open task run benefited significantly from using a larger training corpus.

Limitations and Future Work

  • Limitations:
    • The paper notes that the organization name recognizer in the MSR open task, while identifying many correct names, also generated many false positives, offsetting some of its positive impact.
  • For the CITYU open task, the initial run was less performant because the user dictionary feature was not enabled.
  • The performance of the open run using the CITYU corpus would have been substantially better if the user dictionary had been correctly utilized.
  • Future Work: Not explicitly detailed in the provided text, but the success of the pre- and post-processing modules suggests further exploration in these areas. The authors also mention that a colleague's submission using a different system achieved competitive results, implying that other architectural choices might also be fruitful.

Summary updated

BibTeX
@inproceedings{chen2005unigram,
  author = {Aitao Chen and Yiping Zhou and Anne Zhang and Gordon Sun},
  title = {{Unigram Language Model for Chinese Word Segmentation}},
  booktitle = {Sighan@ijcnlp},
  year = {2005},
}
Figure