Bayesian Semi-Supervised Chinese Word Segmentation for Statistical Machine Translation

Jia Xu, Jianfeng Gao, Kristina Toutanova, Hermann NeyCOLING 2008 · OpenReview

Introduction

Statistical Machine Translation (SMT) from Chinese typically requires an initial word segmentation step, as Chinese text lacks explicit word boundaries. Conventional approaches rely on off-the-shelf Chinese Word Segmentation (CWS) tools trained on manually annotated corpora. However, these "linguistically correct" segmentations often fail to align with the optimal units for translation, particularly when dealing with out-of-vocabulary (OOV) words or domain-specific terminology.

This paper presents a Bayesian semi-supervised model that learns to segment Chinese text specifically to maximize translational utility. By leveraging both monolingual patterns and bilingual alignment data, the proposed method discovers a task-optimal lexicon that improves downstream SMT performance.

Methodology

The core of the approach is a generative model based on a Unigram Dirichlet Process (DP).

Monolingual Word Model

The model treats word generation as a Chinese Restaurant Process (CRP). In this framework, each word is viewed as a "customer" sitting at a "table" representing a word type. This naturally balances the use of frequent, known words with the discovery of new words, controlled by a concentration parameter $\alpha$. A Poisson-based spelling model ensures that newly discovered words maintain statistically plausible lengths.

Bilingual Integration

To ensure segmentations are MT-friendly, the authors incorporate bilingual information via a log-linear framework. This includes an "inverse" IBM Model 1, which encourages segmentations that map consistently to English words. The model considers both Chinese-to-English and English-to-Chinese directions to capture robust alignment features.

Inference via Gibbs Sampling

Given the complexity of the joint segmentation and alignment space, the authors employ Gibbs sampling. This Markov Chain Monte Carlo (MCMC) method iteratively re-samples word boundaries and alignments, allowing the model to converge on an effective segmentation without requiring an exhaustive search.

Experimental Setup

The model was evaluated on two tracks:

  1. NIST 2005 (Large Track): A large-scale news dataset with ~2 million parallel sentences. Due to computational constraints, only the monolingual Bayesian model was applied.
  2. IWSLT 2007 (Small Track): A travel-domain dataset with ~43,000 sentences, utilizing the full bilingual model.

Results and Analysis

Large-Scale Performance (NIST)

On the NIST datasets, the Bayesian model (GS) consistently outperformed the standard LDC baseline. As shown in Figure 3, the method achieved absolute BLEU score improvements of up to 0.41% (NIST 2005). This suggests that a flexible, data-driven lexicon is superior to a fixed manual one, even in high-resource scenarios.

BLEU scores on NIST datasets comparing LDC baseline and Gibbs Sampling (GS).

Small-Scale Performance (IWSLT)

In the IWSLT track, the full bilingual model significantly outperformed established segmenters like ICTCLAS and the LDC tool. On the evaluation set, the Bayesian method reduced the Word Error Rate (WER) by approximately 1.1% compared to the baseline and improved BLEU scores (see Figure 5). Notably, the model achieved a WER of 45.9, matching the best-performing ICT baseline while providing a more principled framework for OOV handling.

Translation performance on IWSLT datasets across various metrics and segmentation methods.

Discussion and Conclusion

The results highlight that the "best" segmentation for SMT is task-dependent rather than purely linguistic. By using a Dirichlet Process prior, the model successfully discovers new words not present in initial dictionaries. Furthermore, bilingual data acts as a powerful disambiguator; characters that consistently map to a single English unit are likely to be merged into a single Chinese word.

While effective, the primary limitation remains the computational cost of Gibbs sampling, which restricted the use of the full bilingual model on the largest datasets. Future work involves exploring more efficient inference techniques and more complex alignment models beyond IBM Model 1 to further refine the segmentation process.

Summary updated

BibTeX
@inproceedings{xu2008bayesian,
  author = {Jia Xu and Jianfeng Gao and Kristina Toutanova and Hermann Ney},
  title = {{Bayesian Semi-Supervised Chinese Word Segmentation for Statistical Machine Translation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2008},
  url = {https://openreview.net/forum?id=q41N3XUymC},
}
Figure