Towards Reliable Neural Machine Translation with Consistency-Aware Meta-Learning

Rongxiang Weng, Qiang Wang, Wensen Cheng, Changfeng Zhu, Min Zhang

Abstract

Neural Machine Translation (NMT) systems often exhibit brittleness, where minor variations in source sentence phrasing lead to inconsistent or poor-quality translations. This paper presents Consistency-Aware Meta-Learning (CAML), a framework designed to enhance NMT reliability. By employing a bilevel optimization strategy, CAML ensures that semantically equivalent inputs are mapped to consistent, high-quality target outputs. Experimental results across multiple benchmarks, including NIST and WMT tasks, demonstrate that CAML significantly outperforms standard Transformer models and improves robustness to source-side diversity.

Introduction

The "source diversity problem" is a significant hurdle for current NMT systems. While natural language allows for numerous ways to express a single concept (a many-to-one mapping), standard training paradigms typically treat translation as a one-to-one mapping between specific source and target pairs. Consequently, models often fail to generalize when faced with lexical or syntactic variations. For instance, substituting a synonym can lead to catastrophic translation failures. This lack of reliability limits the deployment of NMT in professional and mission-critical environments.

Consistency-Aware Meta-Learning (CAML)

To address this, the authors propose CAML, which leverages Model-Agnostic Meta-Learning (MAML) to treat "learning to be consistent" as a meta-task. The framework operates through a bilevel optimization loop:

  1. Outer Loop (Meta-Representation): The model is trained to generate a "consistent meta-representation." It processes a set of semantically equivalent sentences to find a shared hidden state that captures the core meaning, independent of specific wording.
  2. Inner Loop (Translation Mapping): The model then learns to map this stable meta-representation to the correct target output.
The CAML framework illustrates the transition from diverse input embeddings to a unified contextual representation, which is then decoded into consistent output embeddings.

The framework incorporates two primary training objectives:

  • Sentence-level Consistency ($L_S$): A self-supervised task where the model reconstructs a sentence using the representation of its semantically equivalent counterpart.
  • Word-level Consistency ($L_W$): This objective constrains the output probability distributions of equivalent sentences to remain similar at each decoding step.

Training data is augmented using word replacement via alignment models and noisy round-trip translation to generate the necessary semantically equivalent sentence pairs.

Experimental Evaluation

The researchers evaluated the proposed model, CONMT, on several high-profile benchmarks, including NIST Chinese-to-English (Zh→En), WMT tasks (En→De, Zh→En, Ro→En), and the TED M2O multilingual task.

Key Results

The CAML-enhanced models consistently outperformed vanilla Transformer baselines. On the NIST Zh→En task, CONMT achieved a BLEU score of 48.89, representing an improvement of over 3.1 BLEU points compared to the standard Transformer.

Beyond standard benchmarks, the model demonstrated:

  • Robustness to Noise: In simulations with increased input diversity, CAML-trained models maintained significantly higher performance than standard models.
  • Efficiency in NAT: When applied to Non-Autoregressive Translation (NAT) models, CAML improved accuracy by +1.07 BLEU without impacting decoding speed.
  • Scalability: The framework proved effective when scaled to larger architectures like mBART and across multilingual settings.

Discussion and Conclusion

CAML effectively decouples "meaning" from "form" by separating the understanding of core concepts from the generation of target text. By treating consistency as a regularizer, the framework forces the model's internal geometry to remain stable, preventing it from losing semantic consistency during the learning process.

While highly effective, the approach depends on the quality of data augmentation. If augmented sentences alter the original meaning, the model may learn incorrect representations. Additionally, the nested optimization loops of meta-learning increase computational overhead. Future research could extend this focus to target-side diversity to further enhance the naturalness and variety of valid translations.

Summary updated

Abstract

Neural machine translation (NMT) has achieved remarkable success in producing high-quality translations. However, current NMT systems suffer from a lack of reliability, as their outputs that are often affected by lexical or syntactic changes in inputs, resulting in large variations in quality. This limitation hinders the practicality and trustworthiness of NMT. A contributing factor to this problem is that NMT models trained with the one-to-one paradigm struggle to handle the source diversity phenomenon, where inputs with the same meaning can be expressed differently. In this work, we treat this problem as a bilevel optimization problem and present a consistency-aware meta-learning (CAML) framework derived from the model-agnostic meta-learning (MAML) algorithm to address it. Specifically, the NMT model with CAML (named CoNMT) first learns a consistent meta representation of semantically equivalent sentences in the outer loop. Subsequently, a mapping from the meta representation to the output sentence is learned in the inner loop, allowing the NMT model to translate semantically equivalent sentences to the same target sentence. We conduct experiments on the NIST Chinese to English task, three WMT translation tasks, and the TED M2O task. The results demonstrate that CoNMT effectively improves overall translation quality and reliably handles diverse inputs.
BibTeX
@inproceedings{weng2023towards,
  author = {Rongxiang Weng and Qiang Wang and Wensen Cheng and Changfeng Zhu and Min Zhang},
  title = {{Towards Reliable Neural Machine Translation with Consistency-Aware Meta-Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26606},
}
Figure