Morphological Word Segmentation on Agglutinative Languages for Neural Machine Translation

Yirong Pan, Xiao Li, Yating Yang, Rui DongarXiv 2020 · OpenReview

TL;DR

This paper tackles the challenge of translating morphologically rich, agglutinative languages (like Turkish and Uyghur) with Neural Machine Translation (NMT). They propose a novel "morphologically motivated word segmentation" method that combines morpheme analysis with Byte Pair Encoding (BPE) to better represent complex words, significantly improving translation quality by reducing data sparsity and preserving linguistic information.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) models typically use a fixed-size vocabulary to manage computational resources. This approach, however, leads to a significant "Out-of-Vocabulary" (OOV) problem, especially for languages with complex morphology. Agglutinative languages, like Turkish and Uyghur, are characterized by words formed by concatenating multiple morphemes (stems and suffixes). This results in a vast number of word forms, many of which are rare or unseen during training. Consequently, NMT models struggle to effectively capture the linguistic and semantic nuances of these languages, leading to poorer translation quality. Existing methods like simple morpheme segmentation or standard BPE either fail to fully capture morphological richness or ignore morpheme boundaries, respectively.

Methods

The authors propose two new segmentation strategies:

  • BPE on Stem with Combined Suffix (BPE-SCS): This method first segments words into a stem and a combined suffix unit (using "##" for stem and "$$" for suffix). Then, Byte Pair Encoding (BPE) is applied to the stem unit to further break it down into subword units. * **BPE on Stem with Singular Suffix (BPE-SSS):** Similar to BPE-SCS, this strategy segments words into a stem and individual suffix units (using "##" for stem and "$$" for each singular suffix). BPE is then applied to the stem unit.

These methods aim to leverage the strengths of both morpheme segmentation (preserving morphological information) and BPE (reducing vocabulary size and handling rare words) while mitigating their individual weaknesses. They also introduce specific symbols ("##", "$$", "@@") to help the NMT model understand morpheme boundaries and subword units.

Datasets

The experiments were conducted on the following datasets:

  • Turkish-English: WIT corpus and SETimes corpus for training, newsdev2016 for validation, and newstest2016/2017 for testing.
  • Uyghur-Chinese: News data from the China Workshop on Machine Translation (CWMT2017) for training, validation, and testing.

Key Insights and Ideas

  • Morphology Matters for NMT: The paper strongly reinforces the idea that explicitly handling morphological complexity is crucial for achieving high-quality NMT, especially for agglutinative languages.
  • Combining Strengths: The core insight is that a hybrid approach, combining the linguistic awareness of morpheme segmentation with the vocabulary reduction power of BPE, yields superior results compared to using either method in isolation.
  • Stem-Focused BPE: Applying BPE specifically to the stem unit, rather than the entire word or all morphemes, seems to strike a good balance between capturing essential word structure and managing vocabulary size.
  • Symbolic Guidance: The use of special symbols to delineate morpheme and subword boundaries provides valuable structural information to the NMT model.

Key Results

The proposed BPE-SCS and BPE-SSS strategies consistently outperformed baseline methods (Raw, SCS, SSS, and pure BPE) on both Turkish-English and Uyghur-Chinese translation tasks.

  • Turkish-English: BPE-SSS achieved the best results, with improvements of up to 1.2 BLEU points over the strong BPE baseline.
  • Uyghur-Chinese: BPE-SSS also led the pack, showing an improvement of up to 2.5 BLEU points over the BPE baseline.

The paper also explored the impact of the number of merge operations for BPE-SSS, finding that specific ranges (e.g., 25K for Turkish, 30K-35K for Uyghur) maximized translation performance, suggesting an optimal vocabulary size for effective generalization.

Limitations and Future Work

  • Data Domain Mismatch: For Turkish-English, the authors note that the improvement might be limited due to a mismatch between the training (talk/news) and testing (news) data domains, particularly with informal sentences in the training data.
  • Future Work: The authors plan to further incorporate linguistic and morphological knowledge into NMT to enhance its ability to capture syntactic structure and semantic information in low-resource, morphologically-rich languages. They also aim to explore more sophisticated ways to integrate this knowledge.

Summary updated

Abstract

Neural machine translation (NMT) has achieved impressive performance on machine translation task in recent years. However, in consideration of efficiency, a limited-size vocabulary that only contains the top-N highest frequency words are employed for model training, which leads to many rare and unknown words. It is rather difficult when translating from the low-resource and morphologically-rich agglutinative languages, which have complex morphology and large vocabulary. In this paper, we propose a morphological word segmentation method on the source-side for NMT that incorporates morphology knowledge to preserve the linguistic and semantic information in the word structure while reducing the vocabulary size at training time. It can be utilized as a preprocessing tool to segment the words in agglutinative languages for other natural language processing (NLP) tasks. Experimental results show that our morphologically motivated word segmentation method is better suitable for the NMT model, which achieves significant improvements on Turkish-English and Uyghur-Chinese machine translation tasks on account of reducing data sparseness and language complexity.
BibTeX
@misc{pan2020morphological,
  author = {Yirong Pan and Xiao Li and Yating Yang and Rui Dong},
  title = {{Morphological Word Segmentation on Agglutinative Languages for Neural Machine Translation}},
  year = {2020},
  eprint = {2001.01589},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.01589},
}
Figure