Subword-based Tagging by Conditional Random Fields for Chinese Word Segmentation

Ruiqiang Zhang, Genichiro Kikui, Eiichiro SumitaNAACL 2006

TL;DR

This paper introduces a novel "subword-based IOB tagging" approach for Chinese word segmentation, which significantly improves performance over traditional character-based methods by treating frequent multi-character words as single units. They further enhance this by developing a "confidence-dependent" segmentation method that intelligently combines dictionary-based and subword-based tagging, achieving state-of-the-art results on benchmark datasets.

What Problem Does This Paper Address?

Chinese word segmentation is a crucial task in Natural Language Processing, but it's challenging because Chinese doesn't use explicit word delimiters like spaces. Traditional methods often rely on character-based tagging (like IOB - Inside, Outside, Beginning) or dictionary lookups. While dictionary-based methods can achieve high in-vocabulary (IV) rates, they struggle with out-of-vocabulary (OOV) words. Conversely, character-based IOB tagging can handle OOVs better but often results in a lower IV rate and can be less efficient. This paper aims to bridge this gap by proposing a more robust and accurate segmentation method that balances IV and OOV rates and improves overall performance.

Methods

The paper proposes two main innovations:

  • Subword-based IOB Tagging: Instead of tagging each character individually, this method identifies frequent multi-character words from a lexicon and treats them as "subwords." These subwords, along with individual characters, are then tagged using the IOB scheme. This allows the model to recognize common phrases as single units, improving segmentation accuracy and efficiency. Conditional Random Fields (CRFs) are used to train this subword-based tagger.
  • Confidence-Dependent Word Segmentation: This approach addresses the trade-off between dictionary-based (high IV, low OOV recognition) and subword-based IOB tagging (lower IV, higher OOV recognition). It calculates a "confidence measure" for each segmentation decision, combining scores from both the dictionary-based and the subword-based IOB tagging methods. A threshold is then applied: if the confidence is high, the subword-based IOB tag is used; otherwise, the dictionary-based segmentation is preferred. This allows for a flexible trade-off between IV and OOV rates based on user requirements.

Datasets

The experiments were conducted using data from the Sighan Bakeoff 2005 benchmark. This includes four corpora:

  • Academia Sinica (AS)
  • City University of Hong Kong (CITYU)
  • Peking University (PKU)
  • Microsoft Research in Beijing (MSR)

The paper focuses on the "closed test" setting, meaning only the provided training data was used for model training.

Key Insights and Ideas

  • Subwords are Key: Treating common multi-character sequences as single units (subwords) significantly boosts segmentation performance compared to purely character-based approaches. This is because it captures semantic units more effectively.
  • IOB Tagging Benefits from Subwords: The IOB tagging framework, when applied to subwords, proves more powerful than when applied to individual characters.
  • Confidence is Crucial for Trade-offs: The confidence-dependent approach is a clever way to reconcile the strengths and weaknesses of different segmentation strategies (dictionary vs. IOB tagging). It allows for fine-tuning the balance between recognizing known words and handling unknown ones.
  • CRFs are a Powerful Tool: Conditional Random Fields are shown to be effective for learning the complex dependencies in subword-based IOB tagging.

Key Results

  • The proposed subword-based IOB tagging achieved higher F-scores on the CITYU and MSR corpora compared to character-based IOB tagging. For example, F-scores increased from 0.941 to 0.946 for CITYU and from 0.959 to 0.964 for MSR.
  • The confidence-dependent segmentation approach further improved results, achieving state-of-the-art performance on the benchmark datasets. The paper claims the highest F-scores in CITYU, PKU, and MSR corpora compared to other methods from the Sighan Bakeoff 2005.
  • The subword-based approach also offered a speedup in both training and testing due to fewer units being labeled.

Limitations and Future Work

  • Limited Lexicon: The subword-based approach relies on a pre-defined lexicon of frequent words. The effectiveness might be limited if this lexicon is not comprehensive or if the definition of "frequent" is not optimal.
  • Handling of Specific Cases: The paper notes that inconsistent errors can occur with foreign names or Arabic/Chinese numbers if these are not explicitly handled or if they appear inconsistently between training and test sets.
  • Future Work: The authors suggest that incorporating information about Arabic and Chinese numbers, as well as alphabetical characters, could further improve results. They also hint at the potential for yielding better results by using such information to fix inconsistent errors. The confidence measure approach is highlighted as a flexible method that can be tuned to user requirements for R-oov and R-iv rates.

Summary updated

BibTeX
@inproceedings{zhang2006subword,
  author = {Ruiqiang Zhang and Genichiro Kikui and Eiichiro Sumita},
  title = {{Subword-based Tagging by Conditional Random Fields for Chinese Word Segmentation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure