Pre-training via Leveraging Assisting Languages for Neural Machine Translation

Haiyue Song, Raj Dabre, Zhuoyuan Mao, Fei Cheng, Sadao Kurohashi, Eiichiro SumitaACL 2020

TL;DR

This paper tackles the challenge of low-resource Neural Machine Translation (NMT) by proposing a novel pre-training strategy that leverages monolingual data from "assisting" languages. By employing script mapping and careful data selection, they demonstrate significant improvements in Japanese-English translation quality, even when parallel and monolingual data for the target languages are scarce.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) systems typically perform best when trained on large amounts of parallel (bilingual) data. However, many language pairs, especially those considered "low-resource," lack sufficient parallel corpora. While monolingual data can help compensate for this scarcity, it too might be unavailable for the languages of interest (LOI). This paper addresses this dual scarcity by exploring how to effectively utilize monolingual data from other languages (assisting languages) to pre-train NMT models for low-resource scenarios. The core challenge is to bridge the gap between assisting and target languages, especially when they have different scripts and linguistic structures.

Methods

The proposed method builds upon existing sequence-to-sequence (S2S) pre-training techniques like MASS (Masked Sequence to Sequence Pre-training). The key innovations lie in:

  • Leveraging Assisting Languages: Instead of relying solely on LOI monolingual data, the authors utilize larger monolingual corpora from related or unrelated languages.
  • Script Mapping: To increase the similarity between assisting and LOI corpora, they employ script mapping. For Chinese and Japanese, which share characters (Kanji), they use a character-level one-to-one mapping. This aims to increase the number of shared cognates and reduce vocabulary divergence.
  • Data Selection: To mitigate domain mismatch between pre-training and fine-tuning data, they implement two data selection strategies:
  • LM-based Data Selection: Sentences from assisting language corpora are scored using a language model trained on in-domain data, and the highest-scoring sentences are selected.
  • Length-based Data Selection: Sentences from monolingual corpora are selected to match the sentence length distribution of the target language's parallel development set.
  • Pre-training and Fine-tuning: The NMT model is first pre-trained on the processed monolingual data from assisting languages (and LOI if available) using the MASS task. Subsequently, the pre-trained model is fine-tuned on the available parallel data for the target language pair.

Datasets

  • ASPEC Ja-En Parallel Corpus: Used for fine-tuning and evaluation. It's a standard benchmark for Japanese-English translation.
  • Common Crawl Monolingual Corpora: Used for pre-training. These are large, general-domain monolingual datasets.
  • News Commentary Datasets: Used to train language models for data selection, providing a more in-domain source for assisting languages.

Key Insights and Ideas

  • Assisting languages are beneficial: Even unrelated languages can provide some improvement, suggesting that any additional monolingual data can be helpful.
  • Language relatedness matters significantly: While unrelated languages offer some gains, using related languages (like Chinese and Japanese) with script mapping yields substantially better results. This highlights the importance of linguistic similarity.
  • Script mapping is crucial for related languages: Explicitly mapping scripts (like Chinese to Japanese) can artificially increase similarity and boost performance, especially when direct parallel data is scarce.
  • Data selection is key to mitigating domain mismatch: Both LM-based and length-based data selection methods help in making the pre-training data more compatible with the fine-tuning data, leading to better results.
  • Potential for data conflicts: Combining multiple assisting languages can sometimes lead to performance degradation, suggesting that incompatibilities can arise.

Key Results

The paper demonstrates significant improvements in Japanese-English translation quality in low-resource settings:

  • Up to 8.5 BLEU improvement: Achieved by using only Chinese and French monolingual corpora for pre-training in a zero-monolingual data scenario for Japanese and English.
  • Improvements with related languages: Using Chinese (with script mapping) and French as assisting languages consistently outperformed using only Japanese and English monolingual data.
  • Character-level mapping superiority: The one-to-one character-level script mapping between Chinese and Japanese consistently outperformed word-level mapping, achieving about 0.7 to 1.0 BLEU points higher.

Limitations and Future Work

  • Limitations:
    • The study focused on Japanese-English, and the effectiveness might vary for other language pairs.
  • Combining multiple assisting languages sometimes led to performance degradation, indicating potential incompatibilities that need further investigation.
  • The current script mapping is relatively simple; more advanced techniques could be explored.
  • Future Work:
    • Experiment with more challenging language pairs, such as Japanese-Russian.
  • Investigate more rigorous ways to increase relatedness between corpora, potentially using dictionaries and advanced script unification/mapping techniques.
  • Explore subtle relationships among languages to better leverage assisting languages for various NLP tasks.
  • Investigate how to alleviate the observed incompatibilities when combining multiple assisting languages.

Summary updated

BibTeX
@inproceedings{song2020pre,
  author = {Haiyue Song and Raj Dabre and Zhuoyuan Mao and Fei Cheng and Sadao Kurohashi and Eiichiro Sumita},
  title = {{Pre-training via Leveraging Assisting Languages for Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure