Empirical Regularization for Synthetic Sentence Pairs in Unsupervised Neural Machine Translation

Xi Ai, Bin FangAAAI 2021 · DOI

Introduction

Unsupervised Neural Machine Translation (UNMT) has emerged as a powerful paradigm for translating between languages without the need for parallel corpora. Instead, it relies on independent monolingual datasets and a process known as back-translation. In this process, a model translates a sentence from Language A to Language B, and then uses this "synthetic" pair to train the reverse mapping (B to A).

However, back-translation is prone to error, especially in the early stages of training. Low-quality synthetic pairs act as noisy training data, creating a feedback loop that can hinder model convergence and final performance. This paper proposes a suite of regularization losses designed to ensure semantic consistency in synthetic sentence pairs without requiring ground-truth parallel data.

Methodology: Semantic Regularization

The authors introduce a regularization loss, $\mathcal{L}_{reg}$, added to the standard UNMT objective. These losses are categorized into sentence-wise and word-wise approaches.

Sentence-wise Regularization

These methods focus on the global semantic content of the sentence:

  • Auto-encoder Regularization: Employs a 3-layer denoising auto-encoder to extract latent features. The loss minimizes the distance between the latent representations of the original and synthetic sentences.
  • Bag-of-Words (BOW) Regularization: Ensures that the distribution of words in the translation matches the original sentence, regardless of syntax. This is achieved by minimizing the difference between the word-level probability distributions of the two sentences.

Word-wise Regularization

These methods align individual word embeddings:

  • Naive Regularization: Directly maximizes the cosine similarity between word embeddings that are likely translations of each other.
  • GAN Regularization: Uses a Generative Adversarial Network where a discriminator attempts to distinguish between word embeddings of different languages, while the translation model (the generator) tries to align them to fool the discriminator.

To account for differing word orders across languages, the authors utilize a Local Alignment algorithm to reorder synthetic sentences before calculating word-wise losses.

Experimental Results

The proposed methods were evaluated on WMT benchmarks across diverse language pairs, including similar (German, French, Romanian $\leftrightarrow$ English) and dissimilar (Russian $\leftrightarrow$ English) pairs.

Performance Across Pre-trained Models

The regularization techniques were applied to several state-of-the-art UNMT baselines, including XLM, MASS, and mBART. As shown in the results, the addition of GAN or BOW regularization consistently improves BLEU scores across almost all configurations.

BLEU scores for various regularization methods applied to XLM, MASS, and mBART baselines.

Handling Dissimilar Languages

One of the key challenges in UNMT is translating between linguistically distant languages like Russian and English. The experiments demonstrate that while GAN regularization excels on similar language pairs, the Bag-of-Words (BOW) approach is particularly robust for dissimilar pairs. This is likely because BOW does not rely on strict word-to-word alignment, which is harder to achieve for distant languages.

Results for UBWE pre-training, highlighting performance on the Russian-English (Ru-En) task.

Discussion and Insights

  • Semantic Anchoring: Regularization acts as a "semantic anchor," preventing the model from drifting into nonsensical output during the unsupervised phase.
  • Efficiency: Despite the added complexity, these methods are computationally efficient. Training speed is only reduced by 1% to 8%, making them highly practical for real-world research.
  • Hyperparameter Sensitivity: Ablation studies on the regularization weight ($\lambda$) indicate that a balanced weight ($\lambda = 1$) typically yields the best performance, though results remain relatively stable across a range of values.

Conclusion

This work demonstrates that simple yet effective regularization of synthetic sentence pairs can significantly enhance the quality of Unsupervised Neural Machine Translation. By enforcing semantic consistency through sentence-wise and word-wise constraints, the proposed methods mitigate the noise inherent in back-translation, leading to state-of-the-art results across a variety of language pairs and base architectures.

Summary updated

BibTeX
@inproceedings{ai2021empirical,
  author = {Xi Ai and Bin Fang},
  title = {{Empirical Regularization for Synthetic Sentence Pairs in Unsupervised Neural Machine Translation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i14.17479},
}
Figure