Simulated Multiple Reference Training Improves Low-Resource Machine Translation

Huda Khayrallah, Brian Thompson, Matt Post, Philipp Koehn

Abstract

Many valid translations exist for a given sentence, yet machine translation (MT) is trained with a single reference translation, exacerbating data sparsity in low-resource settings. We introduce Simulated Multiple Reference Training (SMRT), a novel MT training method that approximates the full space of possible translations by sampling a paraphrase of the reference sentence from a paraphraser and training the MT model to predict the paraphraser's distribution over possible tokens. We demonstrate the effectiveness of SMRT in low-resource settings when translating to English, with improvements of 1.2 to 7.0 BLEU. We also find SMRT is complementary to back-translation.

Introduction

Standard machine translation (MT) models often struggle in low-resource settings because they are trained to match a single "correct" reference sentence. This approach ignores the vast number of other valid ways to translate the same concept, leading to models that are overly sensitive to specific training examples and fail to generalize. In high-resource settings, models eventually see enough variety to learn linguistic flexibility, but in low-resource scenarios—where data might be limited to a few thousand sentence pairs—this "one-to-one" training approach significantly exacerbates data sparsity.

To address this, we propose Simulated Multiple Reference Training (SMRT). SMRT uses a neural paraphraser to simulate a vast space of possible translations, providing the model with the diversity it lacks in small datasets.

Methods

The core of SMRT is replacing the rigid "one-hot" target of standard training with a soft distribution provided by a pre-trained paraphraser.

The SMRT Objective

  1. The Paraphraser: A Transformer model trained on ParaBank2 (a large dataset of English paraphrases) acts as a teacher.
  2. Soft Targets: Instead of minimizing distance to a single reference word, the MT model is trained to match the paraphraser’s entire probability distribution over the vocabulary. This encourages the model to assign probability to all valid synonyms (e.g., "rabbit", "hare", "bunny").
  3. Sampling the Past: Unlike standard teacher forcing, SMRT samples the previous token from the paraphraser’s distribution. This forces the model to learn how to complete a sentence even if it deviates from the original reference structure.
Figure 1: SMRT approximates the vast space of possible translations by leveraging a paraphraser's distribution to simulate multiple reference paths.

Experimental Setup

We evaluated SMRT on two primary low-resource benchmarks:

  • GlobalVoices: We tested the 10 lowest-resource language pairs with at least 10,000 lines, including languages such as Indonesian, Serbian, and Macedonian.
  • MATERIAL: A benchmark specifically for low-resource settings, focusing on Tagalog-English and Swahili-English bitext.

Results and Analysis

SMRT demonstrated significant performance gains across all tested language pairs, with improvements ranging from 1.2 to 7.0 BLEU points over the baseline.

Figure 3: BLEU scores for SMRT compared to baselines across GlobalVoices and MATERIAL datasets, showing consistent improvements.

Key Insights

  • The Importance of Sampling: Ablation studies revealed that sampling the previous token is the most critical component of SMRT, as it makes the model robust to different sentence structures.
  • Complementarity to Back-Translation: SMRT works effectively alongside back-translation, the current industry standard for low-resource MT. Combining the two methods yielded additional gains of up to 5.7 BLEU, suggesting they address different aspects of data sparsity.
  • Impact of Scale: Gains were most pronounced in the most resource-constrained environments, such as Indonesian-English (8k lines), which saw a +7.0 BLEU boost.

Conclusion and Future Work

SMRT significantly improves low-resource MT by simulating the natural variability of language. While the method currently incurs a roughly 3x computational overhead during training and requires high-quality paraphrasers for the target language, its success in English-target tasks is a promising step forward. Future work will explore the use of multilingual paraphrasers to extend SMRT to a wider variety of language pairs.

Summary updated

Abstract

Many valid translations exist for a given sentence, yet machine translation (MT) is trained with a single reference translation, exacerbating data sparsity in low-resource settings. We introduce Simulated Multiple Reference Training (SMRT), a novel MT training method that approximates the full space of possible translations by sampling a paraphrase of the reference sentence from a paraphraser and training the MT model to predict the paraphraser's distribution over possible tokens. We demonstrate the effectiveness of SMRT in low-resource settings when translating to English, with improvements of 1.2 to 7.0 BLEU. We also find SMRT is complementary to back-translation.
BibTeX
@inproceedings{khayrallah2020simulated,
  author = {Huda Khayrallah and Brian Thompson and Matt Post and Philipp Koehn},
  title = {{Simulated Multiple Reference Training Improves Low-Resource Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=uqKMtU96js3},
}
Figure