Abstract
Neural Machine Translation (NMT) models often suffer from significant performance degradation when encountering "unseen" domains—topics or styles absent from their training data. This paper introduces a novel Leave-One-Domain-Out (LODO) training strategy designed for domain generalization. By dynamically fusing domain-specific adapters and simulating domain shifts during training, the proposed method enables models to handle unknown domains with improved accuracy (up to +0.8 BLEU) without requiring additional fine-tuning or target-domain data.
Introduction
In real-world applications, NMT models are frequently deployed in environments where the input domain is unknown or highly variable. While domain adaptation (tuning on specific target data) is effective, it is often impractical due to the lack of prior data for every possible niche. The challenge of Domain Generalization lies in retaining the specialized knowledge of known domains while learning to generalize to new ones. Traditional approaches often seek "domain-invariant" features, which can inadvertently strip away the stylistic and terminological nuances essential for high-quality translation. This work explores how to effectively mix domain-specific "flavors" to navigate unseen linguistic territories.
Methodology
The researchers propose a two-stage framework built upon a pre-trained encoder-decoder architecture (e.g., mBART).
1. Domain-Specific Parameter Learning
To capture domain-specific features efficiently, small trainable modules called Adapters are inserted into the Transformer layers. During this phase, the massive pre-trained backbone remains frozen, and only one adapter is trained per "seen" domain (e.g., Legal, Medical, IT). This parameter-efficient approach allows the model to specialize without catastrophic forgetting of its general knowledge.
2. AdapterFusion with LODO Training
To handle unseen domains, the model must learn to combine these specialized adapters. An AdapterFusion layer uses an attention mechanism to assign weights to each domain adapter based on the current input.
3. The Leave-One-Domain-Out (LODO) Strategy
The core innovation is the LODO training strategy. Standard fusion training typically assumes access to the target domain. To simulate an "unseen" domain during training, the authors explicitly disable the matching adapter for a given batch. For instance, when processing "Medical" data, the Medical adapter is turned off, forcing the Fusion layer to represent Medical concepts by interpolating between other available adapters (e.g., Legal or IT). This prepares the model for inference on truly unknown domains where no matching adapter exists.
Experimental Setup
The method was evaluated on three language pairs: English-German (En-De), English-French (En-Fr), and English-Polish (En-Pl).
- Source Domains (Seen): Legal (LAW), IT, Koran, Medical (MED), and Subtitles (SUB).
- Unseen Test Domains: Literature (BOOK), presentation transcripts (TED), COVID-19 technical data (TICO19), and the ROBUST dataset.
Key Results
The LODO-Fusion method consistently outperformed strong baselines, including standard zero-shot evaluation and adversarial domain discriminators.
- Performance Gains: The method achieved an average improvement of up to +0.8 BLEU score on unseen domains compared to zero-shot baselines.
- Efficiency: The approach is highly parameter-efficient; while the mBART backbone contains 610 million parameters, only approximately 37 million parameters (adapters and fusion layers) are trained.
- Base Model Impact: Interestingly, models pre-trained on generic data (WMT) often generalized better than those fine-tuned on multiple specific domains, suggesting that "generic" models retain a flexibility that is beneficial for fusion.
Discussion and Limitations
While effective, the LODO strategy relies on clearly labeled source domains to train the initial adapters. In scenarios where domain boundaries are blurry, the effectiveness of discrete adapters may diminish. Additionally, while parameter-efficient in storage, the use of multiple adapters and a fusion layer introduces a minor computational overhead during inference. Future research may explore scaling this approach to hundreds of domain adapters and refining the fusion mechanism for even more granular domain interpolation.
Summary updated