Subword-Based Tagging for Confidence-Dependent Chinese Word Segmentation

Ruiqiang Zhang, Gen-ichiro Kikui, Eiichiro SumitaACL 2006

Introduction

Chinese Word Segmentation (CWS) is a prerequisite for most Chinese natural language processing tasks because Chinese text lacks explicit word delimiters. Historically, CWS has been approached as a character-level sequence labeling problem using the "IOB" (Inside-Outside-Beginning) tagging scheme. While character-based tagging models, such as those using Conditional Random Fields (CRFs), are effective at identifying Out-of-Vocabulary (OOV) words, they often suffer from lower recall on In-Vocabulary (IV) words compared to traditional dictionary-based methods.

This "seesaw" effect creates a challenge for developers who need both high accuracy on known words and robust discovery of new ones. Furthermore, tagging every character in a long sequence is computationally intensive. This paper proposes a hybrid framework that leverages subword-level information and a confidence-based merging mechanism to address these limitations.

Proposed Methodology

The core innovation is the shift from character-based to subword-based tagging. Instead of treating every character as an atomic unit, the authors identify a lexicon of the most frequent multi-character words (subwords) from the training data. During the tagging process, if a character sequence matches a subword in this lexicon, it is treated as a single unit. This approach reduces the sequence length, thereby increasing processing speed, and incorporates lexical knowledge directly into the statistical model.

Confidence-Dependent Merging

To balance the strengths of dictionary-based and statistical tagging models, the authors propose a multi-stage pipeline:

  1. Dictionary-based Segmentation: An initial pass using a standard N-gram model.
  2. Subword-based Tagging: A parallel pass using the subword-based IOB tagger (implemented via MaxEnt or CRF).
  3. Confidence-Based Disambiguation: A final step where results are merged based on a confidence measure.
The proposed CWS pipeline, illustrating the transition from raw input to subword tagging and final confidence-based disambiguation.

By adjusting a confidence threshold ($t$), the system can decide whether to accept a tagger's OOV prediction or default to the dictionary-based segmentation. This provides a tunable "knob" for downstream applications.

Experimental Evaluation

The methodology was tested on four benchmark corpora from the Sighan Bakeoff 2005: PKU (Peking University), MSR (Microsoft Research), CITYU (City University of Hong Kong), and AS (Academia Sinica). The experiments adhered to "closed test" protocols, using only the provided training data for each corpus.

Performance and Speed

The subword-based approach demonstrated significant advantages over character-based baselines. Notably, the reduction in the number of units to be tagged led to a speed increase of approximately 2x during both training and inference. In terms of accuracy, the CRF-based subword tagger achieved state-of-the-art F-scores on three of the four corpora: PKU (0.951), CITYU (0.950), and MSR (0.971).

Managing the IV/OOV Tradeoff

The confidence measure proved vital for managing the performance tradeoff between known and unknown words. As shown in the experimental results, varying the threshold $t$ allows the model to traverse the curve between high IV recall and high OOV recall.

The relationship between In-Vocabulary recall (R-iv) and Out-of-Vocabulary recall (R-oov) as the confidence threshold $t$ varies from 0 to 1 across different corpora.

Conclusion

By combining the efficiency of subword units with the flexibility of a confidence-based merging framework, the authors provide a robust solution to the CWS problem. The approach not only achieves state-of-the-art accuracy but also offers a practical mechanism for tuning performance based on specific task requirements, such as prioritizing the discovery of new entities in search or maintaining high precision on known vocabulary for machine translation.

Summary updated

BibTeX
@inproceedings{zhang2006subword,
  author = {Ruiqiang Zhang and Gen-ichiro Kikui and Eiichiro Sumita},
  title = {{Subword-Based Tagging for Confidence-Dependent Chinese Word Segmentation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2006},
}
Figure