On the Shortcut Learning in Multilingual Neural Machine Translation

Wenxuan Wang, Wenxiang Jiao, Jen-tse Huang, Shuming Shi, Zhaopeng Tu, Michael LyuOpenReview 2022 · OpenReview

TL;DR

This paper identifies a "shortcut learning" problem in multilingual neural machine translation (MNMT) where models overfit to a single "centric" language, causing them to mistranslate into that language during zero-shot translation. They propose a simple training strategy that leverages model "forgetting" to eliminate this shortcut, significantly improving zero-shot performance without extra cost.

What Problem Does This Paper Address?

Multilingual Neural Machine Translation (MNMT) systems are designed to translate between many languages using a single model, offering efficient deployment and cross-lingual knowledge transfer. A key capability is "zero-shot translation," where the model translates between language pairs it hasn't explicitly seen during training. However, a common issue in zero-shot translation is the "off-target" problem: the model translates into an unintended language. This paper investigates the root cause of this off-target issue, particularly in MNMT systems trained on datasets with a single "centric" language (e.g., English as a pivot for many other languages). They hypothesize that the models learn a shortcut, essentially ignoring the true target language and defaulting to the centric one.

Methods

The authors propose a novel training strategy called Generalization Training. This method modifies the standard MNMT training process by dividing it into two phases:

  1. Standard Training Phase: The model is trained on the full dataset for the majority of steps.
  2. Generalization Training Phase: For the final G steps, the model is only trained on examples where the source language is centric and the target language is non-centric. Crucially, examples where the source is non-centric and the target is centric are excluded.

This approach leverages the observed phenomenon that NMT models tend to "forget" previously learned patterns during training. By removing the problematic (non-centric, centric) mapping examples in the later stages, the model is encouraged to forget this shortcut and better learn the desired zero-shot mappings. The authors also analyze the learning dynamics of MNMT models, observing that shortcut learning typically occurs in the later stages of training. They also investigate the impact of multilingual pretraining, finding it can accelerate and worsen this shortcut learning.

Datasets

The paper uses several datasets for training and evaluation:

  • CCMatrix Datasets:
    • Balanced CC6-X: Six balanced datasets where each of six languages (En, De, Fr, Ro, Ja, Zh) serves as the single centric language.
  • Imbalanced CC16-En: An imbalanced English-centric dataset with 16 languages.
  • Noisy Imbalanced OPUS50-En: An imbalanced English-centric dataset with 50 languages, known to contain some noisy examples where target sentences are in the source language.
  • Flores Validation/Test Set: A multi-way dataset containing 997/1012 sentences translated into 101 languages, used for evaluation to mitigate content bias.

Key Insights and Ideas

  • Off-target is "centric-target" bias: The off-target issue in zero-shot translation is not random; it's a strong tendency to translate into the single centric language used in the training data.
  • Shortcut learning is the culprit: This off-target behavior stems from MNMT models overfitting to a shortcut pattern: mapping from any non-centric language directly to the centric language, ignoring the actual target.
  • Pretraining exacerbates the problem: Surprisingly, standard multilingual pretraining (like mBART), while improving supervised translation, actually worsens zero-shot performance by accelerating and amplifying this shortcut learning.
  • Shortcut learning happens late: The problematic shortcut learning typically emerges in the later stages of model training, making it a good target for intervention.
  • Leveraging "forgetting" works: The proposed "Generalization Training" effectively uses the natural forgetting tendency of neural networks to shed the learned shortcut, leading to improved zero-shot translation.

Key Results

  • The proposed Generalization Training approach consistently and significantly improves zero-shot translation performance across various datasets (balanced, imbalanced, noisy) and model configurations (with/without pretraining).
  • On balanced CC6 datasets, their method reduced the off-target ratio (OTR) from over 90% (for pre-trained models) down to around 2-4% while improving zero-shot BLEU scores.
  • The approach maintains the performance of supervised translation, with only a marginal decline in some cases, which is attributed to the removal of centric-to-non-centric training data during the generalization phase.
  • Compared to other methods like "Residual Removing" and "T-Enc Tagging," their approach either outperforms them individually or achieves comparable results when combined.
  • Qualitative analysis (translation examples) shows that the generalization training helps models quickly forget the erroneous centric-target mappings and produce correct in-target translations.

Limitations and Future Work

  • Marginal supervised performance drop: The paper acknowledges that the generalization training phase, by excluding centric-to-non-centric examples, can lead to a slight performance decrease in supervised translation for those specific directions.
  • Future work: The authors suggest that future research should focus on developing MNMT datasets with multiple centric languages, rather than relying on a single one, to inherently mitigate this shortcut learning problem.

Summary updated

Abstract

In this study, we connect the commonly-cited off-target issues in zero-shot translation to the usage of a single centric language in the training datasets of multilingual neural machine translation (MNMT). By carefully designing experiments on different MNMT scenarios and models, we attribute off-target issues to the overfitting of the shortcut patterns of (non-centric, centric) language mappings. Specifically, the learned shortcut patterns biases MNMT to mistakenly translate non-centric languages into the centric language instead of the expected non-centric language. We analyze the learning dynamics of MNMT and find that the shortcut learning generally occurs at the later stage of model training. Pretraining accelerates and aggravates the shortcut learning via a fast transformation from the copy pattern embedded in the pretraining intitialization to the (non-centric, centric) mapping pattern embedded in the MNMT data. Based on these observations, we propose a simple and effective training strategy to eliminate the shortcut patterns in MNMT models by leveraging the forgetting nature of model training. The only difference between our approach and the conventional training is that we only present the training examples of (centric, non-centric) language mapping (excluding the reverse direction) to MNMT models in the later stage of model training. Without introducing any additional data and computational costs, our approach can consistently and significantly improve the performance of zero-shot translation by alleviating the shortcut learning, and maintain the performance of supervised translation for different MNMT models on several benchmarks.
BibTeX
@misc{wang2022shortcut,
  author = {Wenxuan Wang and Wenxiang Jiao and Jen-tse Huang and Shuming Shi and Zhaopeng Tu and Michael Lyu},
  title = {{On the Shortcut Learning in Multilingual Neural Machine Translation}},
  year = {2022},
  url = {https://openreview.net/forum?id=fQGjNpkGVf},
  howpublished = {OpenReview},
}
Figure