Mixed Cross Entropy Loss for Neural Machine Translation

Haoran Li, Wei LuICML 2021

Abstract

Standard Neural Machine Translation (NMT) training treats translation as a one-to-one mapping, forcing models to mimic a single "gold" reference while ignoring valid synonyms. This paper introduces Mixed Cross Entropy (Mixed CE), a loss function that rewards models for predicting both the gold target and their own plausible alternatives. By relaxing the training objective to a one-to-many mapping, Mixed CE significantly improves translation quality and makes models more robust to the "exposure bias" encountered during real-world inference.

The One-to-Many Translation Problem

Machine translation is inherently a one-to-many problem: a single source sentence can be correctly translated using various synonymous words or different sentence structures. However, standard NMT models are typically trained using Cross Entropy (CE) loss, which treats the problem as strictly one-to-one.

CE tries to force the model’s probability distribution to match a "one-hot" vector. This essentially tells the model that only the specific word in the training data is "correct" and everything else—including perfect synonyms like "drop" versus "decline"—is "wrong."

Illustration of the one-to-many translation problem and probability distributions.

This rigidness leads to two main issues:

  1. Poor Generalization: The model fails to learn the rich semantic relationships between synonyms.
  2. Exposure Bias: During training (via "Teacher Forcing"), the model is always fed perfect ground-truth words as input. At test time, it must rely on its own, potentially imperfect, previous predictions. This mismatch between training and testing environments causes errors to compound quickly.

Methodology: Mixed Cross Entropy

The authors propose Mixed Cross Entropy (Mixed CE) as a drop-in replacement for standard CE. The core idea is to augment the loss function with the model's own predictions, treating translation as a "noisy label" problem where the single gold reference is just one representative of a hidden distribution of correct translations.

  • In Teacher Forcing: Instead of just calculating the loss against the gold target, the authors add a second term that calculates the loss against the model's own top prediction. They use a dynamic weight that starts low and increases as training progresses. This allows the model to first learn from the reliable gold data and then "fine-tune" its understanding by exploring its own high-confidence synonyms.
  • In Scheduled Sampling: Scheduled sampling occasionally feeds the model its own predictions during training to simulate test-time conditions. Mixed CE forces the model to produce the same output distribution regardless of whether it was fed the "gold" input or its own "mixed" input. This encourages consistent internal behavior, effectively smoothing out the discrepancy between training and inference.

Unlike Label Smoothing, which spreads probability mass uniformly across the entire vocabulary (including irrelevant words), Mixed CE concentrates probability mass only on plausible alternatives identified by the model itself.

Experimental Setup

The researchers evaluated their method on three standard benchmarks of varying sizes:

  • WMT’16 Romanian-English (Ro-En): 610K sentence pairs (Small).
  • WMT’16 Russian-English (Ru-En): 2.1M sentence pairs (Medium).
  • WMT’14 English-German (En-De): 4.5M sentence pairs (Large).

To test generalization beyond simple n-gram matching, they also utilized a Multi-reference set (a subset of WMT’14 En-De with 10 additional human translations per sentence) and a Paraphrased reference set (a challenging version of the WMT’19 En-De test set where references were intentionally paraphrased to differ significantly from the original training style).

Key Results

Mixed CE demonstrated consistent improvements over standard CE and other baseline loss functions across all evaluated datasets.

Table showing BLEU score improvements across Ro-En, Ru-En, and En-De datasets.
  • Consistent BLEU Gains: On WMT’16 Ro-En, Mixed CE improved the BLEU score from 30.63 to 31.17. Similar gains were observed in the medium and large datasets.
  • Superiority on Paraphrases: On the challenging paraphrased test set, Mixed CE showed a significant lead (11.60 vs. 11.26 BLEU). This confirms the model is learning semantic meaning rather than just memorizing specific word sequences.
  • Multi-Reference Success: In the 10-reference test, Mixed CE consistently generated hypotheses that aligned better with a variety of human translations, showing an average improvement of +0.64 BLEU.

Limitations and Future Work

While highly effective, Mixed CE does introduce some computational overhead. The "two-pass" requirement for scheduled sampling increases training time compared to standard teacher forcing.

Future work could explore applying Mixed CE to non-autoregressive translation models, where generation speed is higher but capturing one-to-many mappings is notoriously difficult. Additionally, investigating how this loss function performs under domain shift (e.g., training on news and testing on medical texts) remains a promising avenue for research.

Summary updated

BibTeX
@inproceedings{li2021mixed,
  author = {Haoran Li and Wei Lu},
  title = {{Mixed Cross Entropy Loss for Neural Machine Translation}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
}
Figure