Multi-Source Transformer for Kazakh-Russian-English Neural Machine Translation

Patrick Littell, Chi-kiu Lo, Samuel Larkin, Darlene StewartACL 2019

TL;DR

This paper tackles the challenge of translating Kazakh to English when parallel data is scarce, but abundant parallel data exists between Kazakh and Russian, and Russian and English. They propose a novel multi-source Transformer architecture that leverages both the original Kazakh and its Russian translation as input, significantly outperforming traditional pivoting and single-source augmentation methods.

What Problem Does This Paper Address?

Machine translation (MT) systems thrive on large amounts of parallel text (sentences and their translations). However, for many language pairs, especially those with fewer speakers or less digital presence, such as Kazakh and English, parallel corpora are small. This paper focuses on the WMT19 Kazakh-English news translation task, where the available Kazakh-English parallel data was limited (~200k sentences). In contrast, there was significantly more parallel data for Kazakh-Russian (~5M sentences) and Russian-English (~14M sentences). The core problem is how to effectively utilize this abundant related-language data to improve translation quality for the low-resource Kazakh-English pair.

Methods

The authors explore three main strategies to leverage the Russian language as a bridge:

  1. Pivoting through Russian: Training separate Kazakh-to-Russian and Russian-to-English MT systems and chaining them.
  2. Synthetic Data Augmentation: Creating synthetic Kazakh-English parallel data by "cross-translating" existing Russian-English data into Kazakh.
  3. Multi-Source Transformer: This is their novel contribution. They adapt the Transformer architecture to accept multiple source inputs simultaneously. Specifically, their system takes both the original Kazakh sentence and its Russian translation as input. This allows the model to attend to information from both sources during the translation process, rather than relying on intermediate, potentially noisy, translations. The multi-source attention mechanism is integrated into the decoder's attention layers, allowing it to draw context from both encoders.

Datasets

  • Kazakh-English: ~200k sentence pairs (news commentary, web-crawled, Wikipedia titles). This was used for training foundation systems for back-translation.
  • Kazakh-Russian: ~5M sentence pairs (web-crawled data). Used for cross-translation.
  • Russian-English: ~14M sentence pairs (web-crawled data, news commentary, Wikipedia titles). Used for back-translation.
  • Development/Evaluation: newsdev2019-kken for tuning and evaluation of Kazakh-English. newstest2017-enru and newstest2018-enru for Russian-English tuning and evaluation.

Key Insights and Ideas

  • Leveraging "Related" Languages is Crucial: When direct parallel data is scarce, using abundant data from related languages (like Russian for Kazakh) is a powerful strategy.
  • Multi-Source Attention Captures More Nuance: Directly feeding multiple source sentences (original and cross-translated) into a multi-source attention mechanism is superior to sequential pivoting or one-way synthetic data generation. This is because the model can attend to the original source directly, avoiding information loss or noise introduced by intermediate translation steps.
  • "Cross-Translation" vs. "Back-Translation": The authors distinguish between "cross-translation" (translating between source languages, e.g., Russian to Kazakh) and "back-translation" (translating from a target language to a source language, e.g., English to Kazakh). This distinction is important for clarity in their pipeline.
  • Transliteration is Key for Joint Models: Converting Kazakh and Russian from Cyrillic to Roman script allows for a unified Byte-Pair Encoding (BPE) model across all three languages, facilitating weight sharing and model efficiency.

Key Results

The proposed multi-source Transformer system achieved state-of-the-art results for the WMT19 Kazakh-English task.

  • It outperformed the pivoting system by 4.2 BLEU.
  • It outperformed augmentation by one-way cross-translation by 10.2 BLEU.
  • In automatic evaluation (BLEU), their system was tied for third place.
  • In human evaluation, they were in a statistical tie for second place.
  • Notably, their system was the top non-ensemble, pure NMT system among the top performers.

Limitations and Future Work

  • Pipeline Complexity: The current approach involves training multiple separate MT systems to generate synthetic data, leading to a complex pipeline.
  • Future Work: The authors suggest exploring ways to use the multi-source architecture itself for creating synthetic data, and sharing encoders/decoders between systems that share languages to reduce complexity. They also aim to investigate other multi-source attention mechanisms and alternative methods for creating cross-linguistic synthetic data beyond machine translation, especially for lower-resource language pairs.

Summary updated

BibTeX
@inproceedings{littell2019multi,
  author = {Patrick Littell and Chi-kiu Lo and Samuel Larkin and Darlene Stewart},
  title = {{Multi-Source Transformer for Kazakh-Russian-English Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure