TL;DR
This paper tackles the challenge of low-resource machine translation for Kazakh-English by employing a "pivoting" strategy. They leverage abundant Russian-language data to create synthetic parallel data, which, when combined with limited existing Kazakh-English data, significantly boosts translation quality using advanced neural networks.
What Problem Does This Paper Address?
Machine translation (MT) systems, especially modern neural MT (NMT) models like the Transformer, thrive on vast amounts of parallel text data (sentences and their translations). However, for many language pairs, such as Kazakh-English, this parallel data is scarce. This scarcity severely limits the performance of NMT systems, leading to poor translation quality. The WMT19 news translation task highlighted this issue with the Kazakh-English language pair. The authors aim to overcome this data bottleneck and achieve competitive translation results for this low-resource scenario.
Methods
The core of their approach is a pivoting technique. Since direct Kazakh-English parallel data is limited, they use Russian as an intermediate language. The process involves:
- Training Pivot SMT Systems: They first train Statistical Machine Translation (SMT) systems for English-Russian and Russian-Kazakh using existing parallel corpora. They opted for SMT for these intermediate systems due to their robustness with limited data and the availability of mature toolkits like Moses.
- Generating Pseudo-Parallel Data: Using the trained pivot SMT systems, they translate the Russian side of the Russian-Kazakh corpus into English (to create English-Kazakh pseudo-parallel data) and the Russian side of the English-Russian corpus into Kazakh (to create Kazakh-English pseudo-parallel data). This effectively "synthesizes" new parallel sentences.
- Combining Data: The generated pseudo-parallel data is then combined with the original, limited parallel Kazakh-English data. They oversample the parallel data (300%) to give it more weight.
- Training NMT Systems: Finally, they train a state-of-the-art attention-based Neural Machine Translation (NMT) model (specifically, a self-attention model based on the Transformer architecture) on this augmented dataset.
They also experimented with subword tokenization (Byte-Pair Encoding or BPE) for both SMT and NMT systems to handle the rich morphology of Kazakh and Russian.
Datasets
The paper utilizes several datasets:
- Kazakh-English: Limited parallel data from sources like News Commentary v14, Wiki Titles v1, and a crawled corpus.
- English-Russian: Extensive parallel data including News Commentary v14, Wiki Titles v1, Common Crawl, ParaCrawl v3, Yandex Corpus, and the United Nations Parallel Corpus v1.0.
- Kazakh-Russian: Parallel data from News Commentary v14 and a crawled corpus.
They specifically mention filtering and sampling from large crawled datasets to create manageable training sets and setting aside development and test sets for evaluation.
Key Insights and Ideas
- Pivoting as a Data Augmentation Strategy: The paper strongly advocates for pivoting as a highly effective method to overcome data scarcity in low-resource MT, especially when a high-resource pivot language is available.
- SMT for Pivot, NMT for Final: They strategically use SMT for the intermediate pivot systems (robustness and ease of training) and then leverage the power of NMT for the final translation task on the augmented data.
- Synthetic Data Quality Matters: The authors note that using synthetic data for the target side of the language pair can degrade quality, reinforcing their approach of generating synthetic data for both directions to create pseudo-parallel corpora.
- Subword Tokenization is Crucial: For morphologically rich languages like Kazakh and Russian, subword tokenization (BPE) is essential for both SMT and NMT to manage vocabulary size and improve generalization.
Key Results
The paper demonstrates significant improvements through their pivoting approach:
- NMT pseudo-p. (final system) achieved a BLEU score of 21.00 for Kazakh-English and 15.47 for English-Kazakh. This is a substantial leap compared to:
- Rule-based MT (RBMT): 1.51 (Kazakh-English), 1.46 (English-Kazakh)
- SMT (word-level): 7.48 (Kazakh-English), 3.82 (English-Kazakh)
- SMT (subword-level): 2.32 (Kazakh-English), 1.42 (English-Kazakh)
- NMT (on parallel data only): 3.53 (Kazakh-English), 1.49 (English-Kazakh)
- The pivot SMT systems themselves achieved respectable scores: 36.05 BLEU for Russian-English and 21.06 BLEU for Russian-Kazakh.
Limitations and Future Work
- Limited Kazakh Monolingual Data: The authors note that while back-translation is a common technique for low-resource scenarios, the available Kazakh monolingual data is limited and of potentially low quality, making it less ideal for extensive use.
- Potential for Further Improvement: They suggest that using extra monolingual English and Kazakh data for the language models of the pivot systems could have yielded further improvements.
- Cyrillic Script Handling: They identified an issue where some sentences remained entirely in Cyrillic script in the final system. Training an additional SMT system specifically for these cases provided a minor BLEU score increase. This suggests that more robust handling of mixed-script or script-identification issues could be beneficial.
Summary updated