Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages

Yunsu Kim, Petre Petrov, Pavel Petrushkov, Shahram Khadivi, Hermann Ney

TL;DR

This paper introduces novel transfer learning techniques for Neural Machine Translation (NMT) between non-English language pairs, leveraging a "pivot" language (like English) to overcome the scarcity of direct parallel data. Their methods significantly improve translation quality, even in low-resource and zero-shot scenarios, by effectively bridging the gap between pre-trained models.

What Problem Does This Paper Address?

Machine translation research has historically focused on language pairs involving English due to the abundance of parallel text data. Translating between two non-English languages (e.g., French to German) often relies on a "pivoting" strategy: translating the source to English, then English to the target. While this works, it doubles decoding time and propagates errors. The core problem this paper tackles is the difficulty of building direct, high-quality NMT systems for non-English language pairs when direct parallel data is scarce. Existing solutions like synthetic data generation or multilingual models have limitations, and this work aims to provide more effective transfer learning strategies.

Methods

The authors propose three novel techniques to improve transfer learning from pre-trained source-pivot and pivot-target NMT models to a direct source-target model:

  • Step-wise Pre-training: This method involves training a single NMT model sequentially. First, a source-pivot model is trained. Then, the encoder from this model is frozen, and a pivot-target model is trained using the frozen encoder and a new decoder. This helps the decoder learn to interpret the source encoder's outputs.
  • Pivot Adapter: This technique adds a small, trainable linear layer (an "adapter") between the pre-trained source encoder and the pre-trained target decoder. This adapter learns a mapping to align the representation space of the source encoder's outputs with what the target decoder is accustomed to from the pivot encoder. It's trained on a small source-pivot parallel corpus.
  • Cross-lingual Encoder: This approach modifies the source-pivot pre-training stage by incorporating an autoencoding objective. The encoder is trained to process both source and pivot language sentences and produce representations in a shared space, facilitating better cross-lingual understanding. Denoising autoencoding is used to encourage learning of linguistic structures.

These methods can be combined for further improvements.

Datasets

The experiments were conducted on WMT 2019 news translation tasks for two non-English language pairs:

  • French → German: Used News Commentary v14 parallel corpus and newstest2008-2010 test sets for fine-tuning.
  • German → Czech: Used a similar setup with available parallel data, including newstest2011 for validation and newstest2012/2013 for testing.

For pre-training, abundant parallel data involving English as a pivot language was used (e.g., French-English, English-German). For synthetic data generation in some experiments, they used pivot-based back-translation to create 5M German-Czech and 9.1M French-German sentence pairs. They also incorporated larger real parallel data (Europarl v7 and Common Crawl) for French-German.

Key Insights and Ideas

  • Bridging the Representation Gap is Crucial: The core challenge in plain transfer learning is that the source encoder is trained for a pivot decoder, and the target decoder for a pivot encoder. The proposed methods directly address this mismatch.
  • Step-wise Training is Surprisingly Effective: Freezing the encoder during the second pre-training step (pivot-target) forces the decoder to adapt to the source encoder's representations, leading to significant gains.
  • Adapters Offer a Compact Solution: The pivot adapter provides a lightweight way to align encoder outputs without retraining the entire encoder, offering a boost with minimal data.
  • Cross-lingual Encoder Benefits from Noise: Using denoising autoencoding for the cross-lingual encoder pre-training is more effective than simple copying, as it forces the model to learn deeper linguistic structures.
  • Zero-Shot Performance is Achievable: The methods demonstrate that effective translation can be achieved even without any direct source-target parallel data, by leveraging pre-trained models and synthetic data.

Key Results

  • The proposed methods significantly outperform baselines like multilingual models and plain transfer.
  • French→German: Achieved up to +2.6% BLEU improvement over multilingual models.
  • German→Czech: Achieved up to +5.2% BLEU improvement over direct baselines in plain transfer.
  • Step-wise pre-training combined with cross-lingual encoder yielded the best results, achieving up to +1.2% BLEU over plain transfer and +2.6% BLEU over multilingual baselines in French-German.
  • In zero-shot scenarios, step-wise pre-training achieved 11.5% BLEU (French-German) and 6.5% BLEU (German-Czech), significantly outperforming plain transfer and even teacher-student models when fine-tuned on synthetic data.
  • The German→Czech zero-shot model achieved 17.2% BLEU, outperforming the best single unsupervised system in WMT 2019.

Limitations and Future Work

  • Pivot Adapter Limitations: The pivot adapter was found to be detrimental when applied after step-wise pre-training, suggesting that the decoder in step-wise pre-training is already well-aligned with the encoder.
  • Cross-lingual Encoder Data Needs: The cross-lingual encoder showed less effectiveness in plain transfer, possibly requiring more data to fine-tune for a new decoder.
  • Future Work: The authors suggest exploring zero-shot translation without step-wise pre-training, by freely combining individually pre-trained encoders and decoders for rapid development of NMT systems for new language pairs.

Summary updated

Abstract

We present effective pre-training strategies for neural machine translation (NMT) using parallel corpora involving a pivot language, i.e., source-pivot and pivot-target, leading to a significant improvement in source-target translation. We propose three methods to increase the relation among source, pivot, and target languages in the pre-training: 1) step-wise training of a single model for different language pairs, 2) additional adapter component to smoothly connect pre-trained encoder and decoder, and 3) cross-lingual encoder training via autoencoding of the pivot language. Our methods greatly outperform multilingual models up to +2.6% BLEU in WMT 2019 French-German and German-Czech tasks. We show that our improvements are valid also in zero-shot/zero-resource scenarios.
BibTeX
@inproceedings{kim2019pivot,
  author = {Yunsu Kim and Petre Petrov and Pavel Petrushkov and Shahram Khadivi and Hermann Ney},
  title = {{Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1080},
}
Figure