TL;DR
This paper tackles the challenge of English-Kazakh (EN-KK) neural machine translation (NMT), a low-resource language pair. The authors explore the benefits of morphological segmentation and leveraging data from related languages (Turkish) and unrelated languages (Russian) through backtranslation, achieving competitive results in the WMT 2019 shared task.
What Problem Does This Paper Address?
Neural Machine Translation (NMT) has revolutionized language translation, but its performance heavily relies on the availability of large, parallel datasets. For many language pairs, such as English-Kazakh (EN-KK), parallel data is scarce, posing a significant challenge for building effective NMT systems. Kazakh, being an agglutinative language, presents additional complexities due to its rich morphology, which standard NMT techniques like Byte-Pair Encoding (BPE) might not fully capture. This paper aims to address these limitations by investigating methods to improve EN-KK translation quality despite data scarcity and morphological complexity.
Methods
The researchers employed several key strategies:
- Morphological Segmentation: They explored both unsupervised (using LMVR, a variant of Morfessor) and rule-based (using Apertium-kaz) morphological segmentation for Kazakh. This was motivated by research suggesting that explicit morphological segmentation can outperform BPE for agglutinative languages.
- Multilingual NMT and Data Augmentation: To combat data scarcity, they leveraged data from related (Turkish) and unrelated (Russian) languages.
- Cyrilization: Turkish, a related language, was transliterated into the Cyrillic script to align with Kazakh's alphabet, enabling its use in multilingual models.
- Backtranslation: This technique was used to generate synthetic parallel data. For KK→EN, they trained a Russian→Kazakh system to translate Russian portions of EN-RU data into Kazakh. For EN→KK, they trained a Russian→English system to translate Russian portions of KK-RU data into English.
- Corpus Filtering: To improve the quality of the training data, they applied several filtering steps, including removing short/long sentences, identifying non-target languages, and filtering based on cosine similarity of sentence embeddings to remove misaligned pairs.
- Model Architecture: The primary NMT architecture used was the Transformer (Vaswani et al., 2017), with the s2s model (Bahdanau et al., 2015) used for experiments with very limited training data.
- Ensembling and Fine-tuning: The best performing models were fine-tuned on the clean EN-KK data, and ensembles of these models were created to further boost performance.
Datasets
The paper utilized several parallel corpora for training and evaluation:
- EN-KK: Kazakhtv, News-comm., Wikititles.
- EN-RU: Common crawl, News-comm., Paracrawl, Yandex.
- KK-RU: Crawl.
- EN-TR: newstest2016-18, SETimes.
They also generated synthetic data through backtranslation, creating new EN-KK' and RU-KK datasets. The evaluation was performed on development sets provided for the WMT 2019 news translation task.
Key Insights and Ideas
- Morphological Segmentation's Potential: While BPE is standard, morphological segmentation (both unsupervised and rule-based) showed promise for Kazakh, achieving comparable or slightly better results than BPE in some configurations, suggesting it can capture linguistic nuances better.
- Leveraging Related and Unrelated Languages: Data from both Turkish (related) and Russian (unrelated) proved beneficial. Backtranslation with Russian, a more abundant resource, yielded significantly larger improvements than using Turkish data alone, highlighting the power of data augmentation even from unrelated languages.
- The Power of Filtering: Aggressive filtering of noisy sentence pairs, particularly using cosine similarity of sentence embeddings, led to substantial improvements in translation quality, underscoring the importance of data cleanliness.
- Synthetic Data's Impact: The addition of synthetic data, especially when carefully filtered, provided a significant boost in performance, demonstrating its effectiveness in low-resource scenarios.
Key Results
- The best EN-KK system achieved a CHRF score of 47.27, and the best KK-EN system reached a BLEU score of 23.11 (fine-tuned BPE model).
- The addition of backtranslated data resulted in massive improvements: +17.9 CHRF points for EN→KK and +14.2 BLEU points for KK→EN.
- Using target synthetic data with a similarity threshold of 0.75 improved EN-KK translation by 1.6 BLEU points.
- The final ensemble systems achieved competitive results, ranking second for Kazakh-English and third for English-Kazakh in the WMT 2019 task.
Limitations and Future Work
- Cyrilization's Negative Impact: Surprisingly, cyrillizing Turkish data led to a decrease in BLEU score for KK→EN translation. The authors plan to investigate this effect further.
- Vowel Harmony: Kazakh exhibits vowel harmony, a linguistic feature that was not explicitly modeled but could potentially improve segmentation and translation quality.
- Synthetic Target Data Exploration: The authors intend to explore the contribution of synthetic target data in more detail.
- Apertium Limitations: The rule-based segmenter (Apertium) faced challenges with out-of-vocabulary words and cases where segmentation did not perfectly reconstruct the original word, leading to lower scores compared to BPE and LMVR.
Summary updated