Prediction Difference Regularization against Perturbation for Neural Machine Translation

Dengji Guo, Zhengrui Ma, Min Zhang, Yang FengACL 2022 · OpenReview

Abstract

Traditional regularization in Neural Machine Translation (NMT) primarily targets over-fitting through "perturb-and-fit" strategies. This paper identifies a significant "under-fitting" phenomenon where models often perform better on noisy inputs than original ones. We propose Prediction Difference Regularization (PD-R), a method that minimizes the gap between a model's predictions on original and perturbed inputs. PD-R addresses both over-fitting and under-fitting, yielding substantial improvements in translation quality and robustness across multiple benchmarks.

Introduction

NMT models are notoriously sensitive to input variations. To improve generalization, researchers typically employ data augmentation or noise injection. The standard assumption has always been that models over-fit the training data, and noise helps them learn more robust features. However, empirical analysis reveals a more complex reality: for a significant portion of training samples (30-40% of tokens), noise actually increases the model's confidence in the correct prediction, indicating a state of under-fitting.

The Under-fitting Revelation

By analyzing the change in prediction probability ($\Delta p$) before and after adding noise, the authors categorized training tokens into two subsets: $S_n$ (over-fitted, where noise decreases confidence) and $S_p$ (under-fitted, where noise increases confidence).

Analysis of over-fitting ($S_n$) and under-fitting ($S_p$) across different perturbation methods.

As shown in Figure 1, the under-fitted subset $S_p$ constitutes a substantial portion of the training data across various perturbation methods like Word-Dropout (WD), Word-Replacement (WR), and Adversarial Training (Adv). Standard regularization methods that force the model to fit noisy data indiscriminately can exacerbate under-fitting in $S_p$, leading to suboptimal performance.

Prediction Difference Regularization (PD-R)

To address this, the researchers propose PD-R. Instead of merely training the model to get the right answer on noisy data, PD-R adds a term to the loss function that minimizes the L1 distance between the probability distributions of the original and perturbed passes.

By forcing the model to produce consistent outputs regardless of small input changes, PD-R acts as a stabilizer. It prevents the model from being too sensitive to specific training examples (mitigating over-fitting) while ensuring it doesn't ignore the nuances of the original data (mitigating under-fitting).

Experimental Results

The method was evaluated on three widely recognized benchmarks: WMT16 English-Romanian (small), WMT16 English-German (medium), and WMT17 Chinese-English (large).

Translation Quality and Robustness

PD-R consistently outperformed vanilla Transformers and established techniques like Word-Dropout, Adversarial Training (AdvT), and R-Drop. On the WMT16 En-De task, PD-R achieved a +1.80 SacreBLEU gain over the baseline. Crucially, the model demonstrated superior robustness when subjected to high levels of noise during inference.

Comparison of BLEU scores across sentence lengths and noise levels.

As illustrated in Figure 2, the PD-R model (Word-Drop+PD-R) maintains higher BLEU scores as the Word-Drop probability increases compared to other methods. Furthermore, it shows particularly strong gains on long, complex sentences (length > 60), which are traditionally the most difficult for NMT models to handle due to exposure bias.

Conclusion

Prediction Difference Regularization offers a more nuanced approach to NMT training by acknowledging the dual nature of fitting issues. While its computational cost is higher due to requiring two forward passes per training step, the gains in translation quality and robustness make it a powerful tool for building more reliable translation systems. Future work will explore applying PD-R to structural or syntactic perturbations beyond word-level changes.

Summary updated

BibTeX
@inproceedings{guo2022prediction,
  author = {Dengji Guo and Zhengrui Ma and Min Zhang and Yang Feng},
  title = {{Prediction Difference Regularization against Perturbation for Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=KDdSyiYI5a},
}
Figure