Towards Enhancing Faithfulness for Neural Machine Translation

Rongxiang Weng, Heng Yu, Xiangpeng Wei, Weihua Luo

Abstract

Neural Machine Translation (NMT) has achieved high levels of fluency, yet it often struggles with faithfulness, leading to omissions or hallucinations. We present FENMT (Faithfulness Enhanced Neural Machine Translation), a novel training strategy designed to improve translation accuracy. FENMT identifies mistranslated fragments during training and utilizes a multi-task learning (MTL) framework to address encoding difficulties, contextual gaps, and language model dominance. Experimental results on WMT benchmarks show significant BLEU score improvements and a substantial reduction in critical translation errors, particularly in long-sentence scenarios.

Introduction

The primary goal of NMT is to produce translations that are both fluent and faithful to the source text. While modern architectures like the Transformer excel at fluency, they are prone to "unfaithfulness"—generating text that sounds natural but omits key source information or includes fabricated details. This paper identifies three root causes: the encoder's inability to represent complex source fragments, the decoder's failure to attend to relevant source context, and the tendency of the decoder's internal language model to override source-driven generation.

Methodology: The FENMT Framework

FENMT introduces a "closed-loop" feedback mechanism that allows the model to learn from its own mistakes during the training process.

Identifying Mistranslated Fragments

Unlike standard "teacher-forcing" training, FENMT incorporates a self-testing phase. At each epoch, the model translates a subset of the training data. These translations are compared against gold-standard references to identify "mistranslated fragments"—phrases that should have appeared in the output but were missed or incorrectly translated.

Multi-Task Learning (MTL) Paradigm

To correct these errors, the model is trained on three auxiliary tasks alongside the standard NMT objective:

  • Encoder Masked Language Modeling (MLM): The system masks source words corresponding to the identified errors. The encoder must then predict these words, forcing it to develop more robust representations of "difficult" input segments.
  • Cross-Attention Alignment: This task guides the decoder's attention mechanism to align more accurately with the source words that correspond to the target fragments, bridging contextual gaps.
  • Max-Margin Objective: To counteract language model dominance, a contrastive learning approach is used. It penalizes the model for generating common, fluent words that are irrelevant to the source, while rewarding the generation of the correct, previously missed fragments.
The FENMT training workflow alternates between standard NMT training and multi-task learning (MTL) based on identified mistranslations.

Experimental Results

The researchers evaluated FENMT on three standard benchmarks: WMT17 Chinese-English (Zh→En), WMT14 English-German (En→De), and WMT16 English-Romanian (En→Ro).

Quantitative and Qualitative Gains

FENMT consistently outperformed the Transformer-Big baseline, achieving a gain of up to 1.44 BLEU points on the Zh→En task. Beyond automated metrics, human evaluation by professional translators revealed that FENMT reduced "Critical" translation errors by 50% and "Major" omissions by 23.8%. These results underscore the model's ability to produce more reliable translations in practice.

Performance on Long Sentences

One of the most significant findings is FENMT's stability when processing long sentences (30-60 words). Standard NMT models often suffer a drop in faithfulness as sentence length increases. In contrast, FENMT maintains high accuracy, as shown in the comparative analysis of BLEU scores across varying sentence lengths.

BLEU score performance across different source sentence lengths, demonstrating FENMT's robustness on longer sequences compared to standard Transformer baselines.

Discussion

A key insight from the ablation studies is the critical role of the encoder; the MLM task provided the most significant performance boost, suggesting that many faithfulness issues originate from poor initial representations. While the training process is slightly slower (approximately 0.86x to 0.94x the speed of standard training), the inference speed remains identical to a standard Transformer, making FENMT highly suitable for production environments.

Conclusion

FENMT offers a practical and effective strategy for enhancing the faithfulness of NMT systems. By actively identifying and addressing its own failures through a multi-task learning curriculum, the model achieves superior accuracy without sacrificing fluency or inference efficiency. Future research will explore the applicability of this "identify-and-fix" paradigm to other text generation tasks such as summarization and dialogue.

Summary updated

Abstract

Neural machine translation (NMT) has achieved great success due to the ability to generate high-quality sentences. Compared with human translations, one of the drawbacks of current NMT is that translations are not usually faithful to the input, e.g., omitting information or generating unrelated fragments, which inevitably decreases the overall quality, especially for human readers. In this paper, we propose a novel training strategy with a multi-task learning paradigm to build a faithfulness enhanced NMT model (named FEnmt). During the NMT training process, we sample a subset from the training set and translate them to get fragments that have been mistranslated. Afterward, the proposed multi-task learning paradigm is employed on both encoder and decoder to guide NMT to correctly translate these fragments. Both automatic and human evaluations verify that our FEnmt could improve translation quality by effectively reducing unfaithful translations.
BibTeX
@inproceedings{weng2020towards,
  author = {Rongxiang Weng and Heng Yu and Xiangpeng Wei and Weihua Luo},
  title = {{Towards Enhancing Faithfulness for Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.212},
}
Figure