Contrastive Preference Learning for Neural Machine Translation

Jianfei He, Shichao Sun, Sen Peng, Jie Xu, Xiaohua Jia, Wenjie Li

Introduction

Standard Neural Machine Translation (NMT) models are typically trained using Maximum Likelihood Estimation (MLE) with teacher forcing. This paradigm, while foundational, introduces a significant discrepancy between training and inference known as exposure bias. During training, the model is conditioned on ground-truth tokens, whereas at inference, it must rely on its own potentially erroneous previous predictions. This often results in hallucinations and a lack of alignment with human-perceived sequence quality. While Reinforcement Learning (RL) techniques have been proposed to mitigate these issues, they are frequently hampered by training instability and high computational costs due to the requirement for online sample generation.

Methodology: Contrastive Preference Learning

We introduce Contrastive Preference Learning (CPL), an offline supervised fine-tuning method designed to align NMT models with human-preferred sequence quality without the overhead of online RL.

Augmented Plackett-Luce Model

CPL extends the Plackett-Luce (PL) model—a standard framework for list-wise ranking—to the domain of sequence generation. By treating translation candidates as a ranked list, the model learns to differentiate between varying levels of quality across multiple hypotheses simultaneously.

The Indicator Function

A key innovation in CPL is the introduction of a "smart" indicator function within the ranking objective. This function acts as a constraint: if the probability margin between a higher-ranked translation and a lower-ranked one already exceeds a predefined threshold, the model ceases to penalize the pair. This mechanism prevents the model from over-optimizing on easy samples, thereby reducing the risk of overfitting and allowing the training process to focus on more challenging, ambiguous cases.

Regularization and Efficiency

To preserve the model's underlying linguistic knowledge, CPL incorporates a token-level cross-entropy regularization term. Because CPL operates on pre-generated offline samples, it achieves a significant speedup—approximately 17 times faster than online RL methods like MIXER.

Experimental Results

The efficacy of CPL was evaluated across three major language pairs: German–English (De–En), Russian–English (Ru–En), and French–English (Fr–En).

Performance and Generalization

CPL consistently outperforms vanilla Transformer models and established baselines such as Scheduled Sampling and Minimum Risk Training. The advantages are most pronounced in out-of-domain (OOD) scenarios, such as medical and biomedical datasets. For instance, on the Ru–En OOD task, CPL achieved a +1.25 BLEU improvement over the baseline. Beyond BLEU, the model showed marked gains in METEOR and COMET scores, indicating better alignment with human judgment.

Training Dynamics

The stability of the CPL training process is evidenced by the convergence of various loss components and the steady improvement of validation metrics over the course of training.

Training dynamics of CPL over 2000 steps, showing (a) validation loss, (b) validation accuracy, (c) token-level cross entropy loss, and (d) DPO loss.

Ablation studies further highlight the importance of the indicator function; removing this component results in a significant performance degradation, confirming its role as the primary driver of the model's robust generalization.

Conclusion

Contrastive Preference Learning offers a compelling alternative to traditional RL for NMT alignment. By combining a list-wise ranking objective with a margin-based indicator function and offline training, CPL achieves superior translation quality and robustness, particularly in challenging out-of-domain contexts, while maintaining high computational efficiency.

Summary updated

Abstract

There exists a discrepancy between the token-level objective during training and the overall sequence-level quality that is expected from the model. This discrepancy leads to issues like exposure bias.To align the model with human expectations, sequence-level objectives are often used to fine-tune pre-trained models.In this paper, we introduce a contrastive preference model that enhances the traditional Plackett-Luce model by incorporating an indicator function. Building upon this novel preference model, we propose Contrastive Preference Learning (CPL), which uses offline samples with list-wise preferences to fine-tune a pre-trained model in Neural Machine Translation. Our experiments, conducted on three language pairs, demonstrate that CPL outperforms not only the vanilla Transformer model but also other token-level and sequence-level baselines. Furthermore, the ablation study highlights the essential role of the proposed indicator function in achieving this improvement.
BibTeX
@inproceedings{he2024contrastive,
  author = {Jianfei He and Shichao Sun and Sen Peng and Jie Xu and Xiaohua Jia and Wenjie Li},
  title = {{Contrastive Preference Learning for Neural Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.174},
}
Figure