Word, Subword or Character? An Empirical Study of Granularity in Chinese-English NMT

Yining Wang, Long Zhou, Jiajun Zhang, Chengqing Zong

Abstract

Neural machine translation (NMT) becomes a new approach to machine translation and is proved to outperform conventional statistical machine translation (SMT) across a variety of language pairs. Most existing NMT systems operate with a fixed vocabulary, but translation is an open-vocabulary problem. Hence, previous works mainly handle rare and unknown words by using different translation granularities, such as character, subword, and hybrid word-character. While translation involving Chinese has been proved to be one of the most difficult tasks, there is no study to demonstrate which translation granularity is the most suitable for Chinese in NMT. In this paper, we conduct an extensive comparison using Chinese-English NMT as a case study. Furthermore, we discuss the advantages and disadvantages of various translation granularities in detail. Our experiments show that subword model performs best for Chinese-to-English translation while hybrid word-character model is most suitable for English-to-Chinese translation.
BibTeX
@inproceedings{wang2017word,
  author = {Yining Wang and Long Zhou and Jiajun Zhang and Chengqing Zong},
  title = {{Word, Subword or Character? An Empirical Study of Granularity in Chinese-English NMT}},
  booktitle = {CWMT},
  year = {2017},
  doi = {10.1007/978-981-10-7134-8\_4},
}

Loading related content...

Figure