Introduction
Neural Machine Translation (NMT) models typically require vast amounts of parallel data to achieve high performance. However, in specialized domains such as medical or legal fields, such data is often scarce. This scarcity leads to "domain divergence," characterized by vocabulary mismatches and polysemy—where words have different meanings depending on context. Standard adaptation techniques like fine-tuning often suffer from overfitting on small target datasets or catastrophic forgetting of general translation knowledge.
To address these challenges, we present MetaMT, a meta-learning framework designed to improve NMT performance in low-resource domains by learning how to adapt efficiently.
Proposed Methodology
MetaMT introduces two primary innovations: domain-invariant word representations and a meta-learning training strategy.
Domain-Invariant Word Representations
Rather than using a static word lookup table, MetaMT projects words into a universal semantic space. A learnable transmission layer ($A^i$) is introduced for each domain to map domain-specific embeddings into this shared space. This mechanism allows the model to resolve polysemy by mapping the same word from different domains to distinct points in the semantic space. To reduce data sparsity, the top 10,000 most frequent words are used as "basis vectors" to define the space.
Meta-Learning Strategy
The training process is inspired by Model-Agnostic Meta-Learning (MAML). Model parameters are split into core translation parameters ($\theta_0$) and meta-parameters ($\theta_1$)—which include the transmission layers and the encoder. The training alternates between "Model Training" on one domain and "Meta Training" on another, forcing the meta-parameters to find an initialization that facilitates rapid adaptation to new domains.
Experimental Setup
The framework was evaluated on English-to-Spanish (En-Es) translation across seven diverse domains, ranging from legal (EU Para, JRC Acquis) and social (Global Voices) to highly specialized fields (Medline, EU Bookshops). Additionally, a custom Electronic Health Record (EHR) dataset of approximately 3,000 sentences was used as a stress test for extreme low-resource scenarios.
Experimental Results
MetaMT consistently outperformed both the standard Transformer baseline and the Transformer with fine-tuning across all tested domains.
Key findings include:
- Significant BLEU Gains: On the JRC Acquis legal dataset, MetaMT achieved 61.04 BLEU, compared to 60.07 for standard fine-tuning.
- Robustness in Low-Resource Settings: On the tiny EHR dataset, MetaMT reached 42.20 BLEU, significantly higher than the 40.61 achieved by fine-tuning and the 36.38 of the base Transformer.
- Ablation Insights: Removing the embedding projection layers (-enc-proj, -dec-proj) resulted in performance degradation, validating the importance of domain-invariant mapping.
Conclusion and Future Work
MetaMT demonstrates that learning the process of adaptation is as critical as learning the translation task itself. While the model shows some limitations with extremely rare words in specific datasets like UN Data, its ability to bridge domain gaps through meta-parameters and universal semantic spaces is a significant step forward for low-resource NMT. Future research may extend this meta-learning approach to other cross-domain NLP tasks such as sentiment analysis or summarization.
Summary updated