Abstract
Neural Machine Translation (NMT) systems often exhibit brittleness, where minor variations in source sentence phrasing lead to inconsistent or poor-quality translations. This paper presents Consistency-Aware Meta-Learning (CAML), a framework designed to enhance NMT reliability. By employing a bilevel optimization strategy, CAML ensures that semantically equivalent inputs are mapped to consistent, high-quality target outputs. Experimental results across multiple benchmarks, including NIST and WMT tasks, demonstrate that CAML significantly outperforms standard Transformer models and improves robustness to source-side diversity.
Introduction
The "source diversity problem" is a significant hurdle for current NMT systems. While natural language allows for numerous ways to express a single concept (a many-to-one mapping), standard training paradigms typically treat translation as a one-to-one mapping between specific source and target pairs. Consequently, models often fail to generalize when faced with lexical or syntactic variations. For instance, substituting a synonym can lead to catastrophic translation failures. This lack of reliability limits the deployment of NMT in professional and mission-critical environments.
Consistency-Aware Meta-Learning (CAML)
To address this, the authors propose CAML, which leverages Model-Agnostic Meta-Learning (MAML) to treat "learning to be consistent" as a meta-task. The framework operates through a bilevel optimization loop:
- Outer Loop (Meta-Representation): The model is trained to generate a "consistent meta-representation." It processes a set of semantically equivalent sentences to find a shared hidden state that captures the core meaning, independent of specific wording.
- Inner Loop (Translation Mapping): The model then learns to map this stable meta-representation to the correct target output.
The framework incorporates two primary training objectives:
- Sentence-level Consistency ($L_S$): A self-supervised task where the model reconstructs a sentence using the representation of its semantically equivalent counterpart.
- Word-level Consistency ($L_W$): This objective constrains the output probability distributions of equivalent sentences to remain similar at each decoding step.
Training data is augmented using word replacement via alignment models and noisy round-trip translation to generate the necessary semantically equivalent sentence pairs.
Experimental Evaluation
The researchers evaluated the proposed model, CONMT, on several high-profile benchmarks, including NIST Chinese-to-English (Zh→En), WMT tasks (En→De, Zh→En, Ro→En), and the TED M2O multilingual task.
Key Results
The CAML-enhanced models consistently outperformed vanilla Transformer baselines. On the NIST Zh→En task, CONMT achieved a BLEU score of 48.89, representing an improvement of over 3.1 BLEU points compared to the standard Transformer.
Beyond standard benchmarks, the model demonstrated:
- Robustness to Noise: In simulations with increased input diversity, CAML-trained models maintained significantly higher performance than standard models.
- Efficiency in NAT: When applied to Non-Autoregressive Translation (NAT) models, CAML improved accuracy by +1.07 BLEU without impacting decoding speed.
- Scalability: The framework proved effective when scaled to larger architectures like mBART and across multilingual settings.
Discussion and Conclusion
CAML effectively decouples "meaning" from "form" by separating the understanding of core concepts from the generation of target text. By treating consistency as a regularizer, the framework forces the model's internal geometry to remain stable, preventing it from losing semantic consistency during the learning process.
While highly effective, the approach depends on the quality of data augmentation. If augmented sentences alter the original meaning, the model may learn incorrect representations. Additionally, the nested optimization loops of meta-learning increase computational overhead. Future research could extend this focus to target-side diversity to further enhance the naturalness and variety of valid translations.
Summary updated