Word, Subword or Character? An Empirical Study of Granularity in Chinese-English NMT

Yining Wang, Long Zhou, Jiajun Zhang, Chengqing Zong

Abstract

Neural machine translation (NMT) has proven to outperform conventional statistical machine translation (SMT) across various language pairs. However, NMT systems typically operate with a fixed vocabulary, leading to an "open-vocabulary" problem where rare words are replaced by unknown symbols. This study conducts an extensive comparison of translation granularities—word, subword, character, and hybrid—specifically for Chinese-English NMT. Our experiments demonstrate that subword models perform best for Chinese-to-English translation with moderate vocabulary sizes, while hybrid word-character models are most suitable for English-to-Chinese translation. Notably, the Hybrid_BPE method achieves the best results on the Chinese-to-English task.

Introduction

Most NMT systems encounter a bottleneck when dealing with "rare words" not present in their fixed vocabulary. These words are often replaced with a generic <UNK> symbol, causing significant information loss and poor translation quality. This issue is particularly challenging in Chinese-English translation due to the lack of explicit word boundaries in Chinese and the complex morphology of English. This paper investigates the optimal granularity—balancing semantic density and vocabulary flexibility—to maximize translation accuracy (BLEU scores) for this specific language pair.

Methods

The study utilizes a standard attention-based encoder-decoder NMT architecture with stacked LSTMs. The core of the investigation lies in comparing different tokenization strategies to find the "Goldilocks" level of granularity:

  • Word Level: The traditional approach using standard segmentation tools.
  • Character Level: Breaking Chinese sentences into individual characters.
  • Hybrid Word-Character: Keeping frequent words intact but breaking rare words into characters. Special prefixes (<B>, <M>, <E>) denote the position of a character within a word to allow for reversible processing.
  • Subword Level: Using data-driven compression algorithms like Byte Pair Encoding (BPE), which merges frequent adjacent sequences, or the Wordpiece Model (WPM), which maximizes training data likelihood.
Comparison of different text segmentation granularities for a Chinese sentence.

Key Results and Insights

The experiments, conducted on 2.09 million sentence pairs from LDC corpora, revealed several critical findings regarding the impact of granularity on translation performance:

  1. Subword Robustness: Subword models (BPE and WPM) are more robust to vocabulary size constraints than word-level models. At a 15k vocabulary size, WPM reached 43.72 BLEU, significantly outperforming the word-level model's 39.97.
  2. Directionality Matters: Subwords excel when translating into English, likely due to their ability to handle English morphology. Conversely, hybrid models are superior for Chinese targets, balancing word-level semantics with character-level flexibility.
  3. Performance on Long Sentences: Subword models showed a marked advantage in translating long sentences (60-80+ words) compared to pure word or character-level approaches.
BLEU scores for various segmentation methods (30k vocabulary) on Chinese-to-English translation tasks.

As shown in the results table, the BPE method achieved the highest average BLEU score of 43.75 for Chinese-to-English translation at a 30k vocabulary size, while the Wordpiece method achieved the highest score on the MT03 development set.

Conclusion

This empirical study highlights that there is no "one-size-fits-all" granularity for NMT. While subword models like BPE are highly effective for English targets, hybrid models offer advantages for Chinese. Interestingly, increasing vocabulary size beyond a certain point (e.g., from 30k to 60k for subwords) can lead to diminishing returns, suggesting that excessive granularity may introduce noise. Future work will explore these strategies in low-resource settings and other logographic language pairs.

Summary updated

Abstract

Neural machine translation (NMT), a new approach to machine translation, has been proved to outperform conventional statistical machine translation (SMT) across a variety of language pairs. Translation is an open-vocabulary problem, but most existing NMT systems operate with a fixed vocabulary, which causes the incapability of translating rare words. This problem can be alleviated by using different translation granularities, such as character, subword and hybrid word-character. Translation involving Chinese is one of the most difficult tasks in machine translation, however, to the best of our knowledge, there has not been any other work exploring which translation granularity is most suitable for Chinese in NMT. In this paper, we conduct an extensive comparison using Chinese-English NMT as a case study. Furthermore, we discuss the advantages and disadvantages of various translation granularities in detail. Our experiments show that subword model performs best for Chinese-to-English translation with the vocabulary which is not so big while hybrid word-character model is most suitable for English-to-Chinese translation. Moreover, experiments of different granularities show that Hybrid_BPE method can achieve best result on Chinese-to-English translation task.
BibTeX
@inproceedings{wang2017word,
  author = {Yining Wang and Long Zhou and Jiajun Zhang and Chengqing Zong},
  title = {{Word, Subword or Character? An Empirical Study of Granularity in Chinese-English NMT}},
  booktitle = {CWMT},
  year = {2017},
  doi = {10.1007/978-981-10-7134-8\_4},
}
Figure