Relevance-guided Neural Machine Translation

Isidora Chara Tourni, Derry WijayaarXiv 2023 · OpenReview

TL;DR

This paper introduces a novel approach to improve Neural Machine Translation (NMT) by incorporating explainability during training. By leveraging Layer-wise Relevance Propagation (LRP) to understand which input tokens contribute most to the output, they guide the model to focus on more relevant information, showing promising results, especially in low-resource scenarios.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) has seen tremendous progress, largely thanks to the Transformer architecture. However, a persistent challenge lies in low-resource settings where limited parallel data hinders performance. Existing methods like Back-Translation and Auto-Encoding help, but there's a gap in leveraging model interpretability to boost translation quality, especially in data-scarce environments. This paper aims to bridge this gap by using explainability to enhance NMT training.

Methods

The core of this work is an "explainability-based training approach" for NMT. They adapt the Layer-wise Relevance Propagation (LRP) technique, originally used for explaining model decisions, to guide the training process. LRP quantifies the contribution of input tokens to the model's output.

The authors modify the standard NMT loss function to include an LRP-guided term. This new loss function, $L'$, is a weighted sum of the original loss $L(y, p)$ and an explanation-guided loss $L(y, p_{lrp})$. The weights $\xi$ and $\lambda$ control the influence of the original and LRP-guided predictions. By reweighting intermediate feature representations with their relevance scores, the model is encouraged to focus on more informative parts of the input, effectively downscaling less relevant information. This approach is applied to both unsupervised and supervised NMT training.

Datasets

The experiments were conducted using the following language pairs:

  • English-French (En-Fr), French-English (Fr-En): 5 million monolingual sentences from News Crawl 2007-2008 and 23 million WMT14 parallel sentences.
  • English-Gujarati (En-Gu), Gujarati-English (Gu-En): 1.4 million monolingual sentences and 22k parallel sentences from WMT 2019 News Translation Task.
  • English-Kazakh (En-Kk), Kazakh-English (Kk-En): 9.5 million monolingual sentences and 132k parallel sentences from WMT 2019.

Test and validation sets included newstest2013/2014 for French and WMT19 for Gujarati and Kazakh.

Key Insights and Ideas

  • Explainability as a Training Signal: The paper demonstrates that insights from model interpretability (specifically LRP) can be directly integrated into the training loop to improve model performance, not just for post-hoc analysis.
  • Low-Resource Potential: The LRP-guided approach shows particular promise in low-resource settings, where data scarcity is a major bottleneck. This suggests that making models more "aware" of what's important can compensate for limited data.
  • Morphological Complexity Hypothesis: The authors hypothesize that LRP-guided training might be more beneficial for languages with complex morphology (like Gujarati), as it could help the model better disentangle and focus on relevant linguistic features.
  • Parameter Sensitivity: Different hyperparameter settings for weighting the LRP-guided loss ($\xi, \lambda$) yielded varying results, indicating that careful tuning is crucial for optimal performance.

Key Results

  • En-Fr/Fr-En: In high-resource settings for English-French, the LRP-guided method did not significantly outperform baseline NMT models, with BLEU score differences often within 0.1-0.5, suggesting it might not be as effective in data-rich scenarios.
  • En-Gu/Gu-En: For English-Gujarati, a notable improvement was observed when using MT-only or BT-AE-MT objectives, with results marginally outperforming previous state-of-the-art approaches.
  • En-Kk/Kk-En: The LRP-guided training significantly improved NMT BLEU scores compared to regular models, especially in low- (22k) and mid-resource (132k) settings. In many cases, these experiments also surpassed existing state-of-the-art results.

Limitations and Future Work

  • Computational Cost: Employing LRP-guidance significantly increases training time and computational resource requirements, necessitating more efficient training solutions.
  • Limited Experiments: The study was conducted with a small number of experiments, limiting the scope for extensive hyperparameter tuning and language expansion.
  • Future Directions: The authors strongly recommend further exploration of the method with different hyperparameter setups, expansion to more languages, and investigation into more efficient training and computational strategies to enhance its efficiency and robustness.

Summary updated

Abstract

With the advent of the Transformer architecture, Neural Machine Translation (NMT) results have shown great improvement lately. However, results in low-resource conditions still lag behind in both bilingual and multilingual setups, due to the limited amount of available monolingual and/or parallel data; hence, the need for methods addressing data scarcity in an efficient, and explainable way, is eminent. We propose an explainability-based training approach for NMT, applied in Unsupervised and Supervised model training, for translation of three languages of varying resources, French, Gujarati, Kazakh, to and from English. Our results show our method can be promising, particularly when training in low-resource conditions, outperforming simple training baselines; though the improvement is marginal, it sets the ground for further exploration of the approach and the parameters, and its extension to other languages.
BibTeX
@misc{tourni2023relevance,
  author = {Isidora Chara Tourni and Derry Wijaya},
  title = {{Relevance-guided Neural Machine Translation}},
  year = {2023},
  eprint = {2312.00214},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.00214},
}
Figure