Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation

Weijia Xu, Xing Niu, Marine Carpuat

Abstract

Exposure bias—the gap between a model seeing perfect "teacher" data during training and its own imperfect predictions at test time—remains a major hurdle for Neural Machine Translation (NMT). This paper introduces Soft Aligned Maximum Likelihood (SAML), a differentiable sampling approach that allows models to learn from their own mistakes while flexibly aligning their predictions to reference translations, leading to significant BLEU score improvements without the need for complex training schedules.

Introduction

Most Neural Machine Translation (NMT) models are trained using "Teacher Forcing." During training, the model predicts the next word given the actual previous words from the reference translation. However, at inference (test time), the model must rely on its own previously generated words, which may contain errors. This discrepancy, known as exposure bias, causes errors to compound rapidly as the model enters states it never encountered during training.

Previous attempts to mitigate this, such as Scheduled Sampling (SS), occasionally feed the model its own predictions during training. However, these methods typically assume a strict 1-to-1 alignment, expecting the $n$-th predicted word to match the $n$-th reference word. In translation, word order is often flexible. If a model predicts a valid translation with a different word order, Scheduled Sampling penalizes it unfairly, providing "misleading training signals."

Soft Aligned Maximum Likelihood (SAML)

The authors propose SAML, a two-pronged solution to address exposure bias while respecting the flexibility of language:

  1. Differentiable Sampling: To enable the model to learn from its mistakes, the training process must be differentiable. SAML utilizes the Straight-Through Gumbel-Softmax estimator. This allows the model to make discrete word choices in the forward pass while using a continuous approximation in the backward pass to update weights via gradient descent.
  2. Soft Alignment: Instead of forcing a rigid word-to-word match, SAML computes a "soft alignment" between the reference translation and the sampled sequence. Using the model's internal hidden states, it determines which predicted words likely correspond to which reference words. The training objective then maximizes the probability that the reference can be aligned with the sampled output.
Comparison of Scheduled Sampling and SAML objectives.

Essentially, the model is taught that producing the correct word is more important than producing it at a specific index, provided the overall sequence remains coherent.

Experimental Results

The approach was evaluated on three standard translation benchmarks from the International Conference on Spoken Language Translation (IWSLT): German-English (de-en), English-German (en-de), and Vietnamese-English (vi-en).

Key Findings

  • Consistent BLEU Gains: SAML outperformed both standard Maximum Likelihood (MLE) and Scheduled Sampling baselines across all tasks, achieving improvements of up to +1.0 BLEU on Vietnamese-English.
  • Robustness to Search Constraints: SAML proved particularly effective when using small beam sizes. This suggests that SAML-trained models are inherently more robust and less dependent on expensive search algorithms to find high-quality translations.
  • Simplicity in Training: Unlike standard Scheduled Sampling, which requires a carefully tuned "annealing schedule," SAML works effectively with a constant sampling rate, making it significantly easier to implement.
BLEU improvement across different beam sizes.

Conclusion

SAML demonstrates that the key to overcoming exposure bias is not just seeing errors, but evaluating them flexibly. By incorporating soft alignment, NMT models can learn more effectively from their own predictions. While the current work focuses on RNN-based architectures, future research could explore how soft alignment interacts with the self-attention mechanisms of Transformer models or investigate more sophisticated alignment functions.

Summary updated

BibTeX
@inproceedings{xu2019differentiable,
  author = {Weijia Xu and Xing Niu and Marine Carpuat},
  title = {{Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=r1bEhQWubr},
}
Figure