Abstract
Neural machine translation (NMT) has achieved remarkable success in producing high-quality translations. However, current NMT systems suffer from a lack of reliability, as their outputs are often affected by lexical or syntactic changes in inputs, resulting in large variations in quality. This limitation hinders the practicality and trustworthiness of NMT. A contributing factor to this problem is that NMT models trained with the one-to-one paradigm struggle to handle the source diversity phenomenon, where inputs with the same meaning can be expressed differently. In this work, we treat this problem as a bilevel optimization problem and present a consistency-aware meta-learning (CAML) framework derived from the model-agnostic meta-learning (MAML) algorithm to address it. Specifically, the NMT model with CAML (named CoNMT) first learns a consistent meta representation of semantically equivalent sentences in the outer loop. Subsequently, a mapping from the meta representation to the output sentence is learned in the inner loop, allowing the NMT model to translate semantically equivalent sentences to the same target sentence. We conduct experiments on the NIST Chinese to English task, three WMT translation tasks, and the TED M2O task. The results demonstrate that CoNMT effectively improves overall translation quality and reliably handles diverse inputs.
Introduction: The Challenge of Source Diversity
Despite the impressive performance of modern Neural Machine Translation (NMT) systems, they remain surprisingly brittle. A core issue is the "one-to-one" training paradigm: models are typically trained to map one specific source sentence to one specific target sentence. In real-world usage, however, the same meaning can be expressed through a vast array of lexical and syntactic variations—a phenomenon known as source diversity.
Standard NMT models often fail to maintain consistency when faced with these variations. A slight change in word choice or sentence structure can lead to significant drops in translation quality or even "hallucinations." To be truly reliable and trustworthy, NMT systems must transition toward a "many-to-one" mapping capability, ensuring that semantically equivalent inputs result in consistent, high-quality outputs.
Methodology: Consistency-Aware Meta-Learning (CAML)
To bridge this gap, we propose the Consistency-Aware Meta-Learning (CAML) framework. This approach treats the translation of semantically equivalent sentences as a series of related "tasks" within a bilevel optimization structure, building upon the Model-Agnostic Meta-Learning (MAML) algorithm.
The training process for our model, CoNMT, is divided into two distinct loops:
- The Outer Loop (Meta-Representation): The goal here is to force the encoder to produce a consistent "meta-representation" for any set of semantically equivalent sentences. To achieve this, we introduce two novel loss functions:
- Sentence-level Loss ($L_S$): A self-supervised objective where the model attempts to reconstruct one equivalent sentence using the representation derived from another.
- Word-level Loss ($L_W$): This ensures that the output probability distributions for different equivalent inputs are as similar as possible at each decoding step.
- The Inner Loop (Mapping): Once a consistent meta-representation is established, the model learns the optimal mapping from this representation to the ground-truth target sentence.
To provide the necessary training data, we utilize a two-step augmentation process involving word replacement (guided by alignment models) and noisy round-trip translation to generate high-quality semantically equivalent source sentences.
Experimental Results
We evaluated CoNMT across several rigorous benchmarks, including NIST Chinese-to-English (Zh-En), multiple WMT tasks (En-De, Zh-En, Ro-En), and the TED Many-to-One (M2O) multilingual task.
Performance and Reliability
CoNMT demonstrated significant improvements over baseline Transformer models. On the NIST Zh-En task, it improved the baseline by up to 3.23 absolute BLEU points. Across all WMT tasks, the model achieved gains of over 1.0 BLEU point.
Crucially, CoNMT exhibits superior reliability. When we increased the proportion of "diverse" (paraphrased) inputs during testing, the performance of the standard Transformer plummeted, while CoNMT maintained a much higher and more stable quality level.
Key Insights
- Consistency as a Regularizer: Our findings suggest that simply adding augmented data is insufficient. Explicitly constraining the model's internal state via $L_S$ and $L_W$ is the "secret sauce" that stabilizes the decoder against input variations.
- Scalability: The CAML framework is architecture-agnostic. We observed significant gains not only on standard Transformers but also when applied to "Deep Transformers" (159M parameters) and massive pre-trained models like mBART (610M parameters). This indicates that increasing model size alone does not solve the source diversity problem; architectural or algorithmic changes like CAML are necessary.
Conclusion
The CoNMT framework addresses a critical vulnerability in modern NMT by enforcing consistency across semantically equivalent inputs. By leveraging meta-learning to create a unified internal representation, we significantly improve both the raw translation quality and the real-world reliability of NMT systems. Future work will explore more sophisticated methods for defining semantic equivalence and extending this consistency framework to other natural language generation tasks.
Summary updated