Abstract: Huda Khayrallah, Brian Thompson, Matt Post, Philipp Koehn. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Introduction
Standard Neural Machine Translation (NMT) models are typically trained to match a single "correct" reference translation. This approach ignores the vast linguistic diversity inherent in language; a single source sentence can often be translated in billions of valid ways. In high-resource settings, the sheer volume of data eventually exposes the model to sufficient variety. However, in low-resource settings (e.g., Tagalog-to-English), this "one-reference-itis" creates a massive data sparsity problem. The model learns to be overly rigid, attempting to replicate a specific string of words rather than learning the underlying mapping of meaning.
SMRT: Simulated Multiple Reference Training
To address this, the authors introduce Simulated Multiple Reference Training (SMRT). Instead of forcing the MT model to predict a single "gold" word at each step, SMRT encourages the model to match the entire distribution of possible words provided by a pre-trained neural paraphraser.
Key components of the approach include:
- Neural Paraphraser: A Transformer model trained on ParaBank2 (a massive dataset of English paraphrases) acts as a teacher.
- Soft Targets: The MT model is trained to minimize the cross-entropy between its output and the paraphraser’s probability distribution, rather than a standard one-hot vector.
- Sampling during Training: Instead of standard teacher forcing, the model samples previous words from the paraphraser’s distribution. This exposes the MT model to a wider variety of sentence structures and forces it to learn how to "recover" and continue a translation even when it deviates from the original reference.
- Hybrid Objective: The final training objective is a 50/50 mix of the SMRT loss and the standard negative log-likelihood (NLL) loss on the original reference.
Experimental Setup
The researchers focused on English as the target language to leverage high-quality English paraphrasers. They tested the method on 12 low-resource language pairs from the MATERIAL Program (Tagalog and Swahili to English) and GlobalVoices (including Hungarian, Indonesian, Serbian, and Arabic to English).
Key Results
SMRT improved performance across all 12 tested language pairs, with gains ranging from 1.2 to 7.0 BLEU points. The most striking improvement was seen in Indonesian-to-English, which jumped from a baseline of 5.3 to 12.3 BLEU.
Ablation studies revealed that sampling is the "secret sauce" of the method. Simply using the paraphraser's distribution in the loss function was insufficient; the real gains came from sampling tokens, which forces the model to explore the space of possible translations. Furthermore, SMRT was found to be complementary to back-translation, the current industry standard for low-resource MT, providing additive benefits when used together.
Discussion and Limitations
While highly effective, SMRT introduces a 3x computational overhead during training because it requires running the paraphraser model in tandem with the MT model. Additionally, the method currently requires a high-quality paraphraser in the target language, making it most applicable for translations into high-resource languages like English. Future work may leverage multilingual paraphrasers to expand the reach of SMRT to a broader array of language pairs.
Summary updated