BUPT Systems in the SIGHAN Bakeoff 2007

Ying Qin, Caixia Yuan, Jiashen Sun, Xiaojie Wang

Introduction

In Chinese Natural Language Processing (NLP), the lack of a singular annotation standard presents a significant challenge. Different corpora, such as the Peking University (PKU) corpus or the Chinese Treebank (CTB), follow distinct rules for word segmentation and part-of-speech (POS) tagging. Consequently, systems often struggle with "robustness"—the ability to maintain performance across varying standards and diverse text domains. The Beijing University of Posts and Telecommunications (BUPT) systems for the 2007 SIGHAN Bakeoff were designed specifically to address these challenges through a modular, multi-stage architecture.

Methodology

The BUPT team developed a pipeline for three primary tasks: Word Segmentation (WS), Named Entity Recognition (NER), and POS tagging.

Word Segmentation (WS)

The WS system employs a three-step process:

  1. Basic Segmentation: Utilizes a forward-backward maximum matching algorithm against a common wordlist of 64,000 words ratified across four different standards.
  2. Disambiguation: Overlapping ambiguities are resolved using a class-based bigram model, refined by a "disambiguation adaptor" to align with specific corpus standards.
  3. OOV Recognition: Out-of-vocabulary (OOV) words are categorized into "factoids" (e.g., dates, numbers), handled by finite-state automata, and "non-factoids," processed via a character-based Conditional Random Field (CRF) model.

Named Entity Recognition (NER)

Two distinct approaches were evaluated for NER:

  • System 1 (CRF-based): A minimalist model relying on a small dictionary and basic rules for coordinate relations.
  • System 2 (Maximum Entropy-based): A more complex approach using individual models for Person, Location, and Organization entities, incorporating heavy-duty, corpus-derived rules for boundary detection.
Comparison of NER performance between System-1 (CRF) and System-2 (Maximum Entropy).

POS Tagging

The team implemented a two-step Maximum Entropy (ME) framework. The first step uses standard contextual syntactic features, while the second step applies a "finer" model to resolve multi-tag ambiguities. By calculating the conditional probability of a tag based on the preceding tag (e.g., if the previous word is a quantifier, a word like "Report" is 91.67% likely to be a noun), the system significantly reduces tagging errors.

Evaluation and Results

The systems were evaluated across several SIGHAN Bakeoff 2007 tracks, including SXU, NCC, PKU, and CTB. The People’s Daily (2000) corpus was utilized for training bigram transition probabilities.

Performance and Encoding Issues

A critical finding was the impact of character encoding on system performance. While the BUPT systems were natively designed for GBK encoding, the official SIGHAN submissions required UTF-16. Conversion errors led to a notable performance dip in some tracks, most significantly in the PKU POS tagging task, where the F-measure dropped from 0.9462 (GBK) to 0.6649 (UTF-16).

POS tagging performance comparison highlighting the impact of encoding (UTF-16 vs GBK).

As shown in the results, the rule-heavy Maximum Entropy system for NER (System 2) outperformed the minimalist CRF approach, achieving an F-measure of 0.8912. In POS tagging, the system demonstrated high stability when using native GBK encoding, reaching an F-measure of 0.9689 for CTB.

Key Insights and Conclusion

The BUPT team's "adaptor" strategy proves that modularity is an efficient alternative to retraining massive models for every new standard. By building a "basic unit" and using adaptors for tuning, the system achieves high flexibility.

However, the authors emphasize that while "standard robustness" (fitting rules) is the focus of current bakeoffs, "corpus robustness" (handling domain shift) remains the more significant hurdle for real-world applications. Future work will focus on improving generalization across diverse text sources and resolving technical hurdles related to cross-encoding compatibility.

Summary updated

BibTeX
@inproceedings{qin2008bupt,
  author = {Ying Qin and Caixia Yuan and Jiashen Sun and Xiaojie Wang},
  title = {{BUPT Systems in the SIGHAN Bakeoff 2007}},
  booktitle = {Ijcnlp},
  year = {2008},
}
Figure