Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training

Thuy-Trang Vu, Shahram Khadivi, Dinh Q. Phung, Gholamreza Haffari

Abstract

Neural Machine Translation (NMT) models often suffer from significant performance degradation when encountering "unseen" domains—topics or styles absent from their training data. This paper introduces a novel Leave-One-Domain-Out (LODO) training strategy designed for domain generalization. By dynamically fusing domain-specific adapters and simulating domain shifts during training, the proposed method enables models to handle unknown domains with improved accuracy (up to +0.8 BLEU) without requiring additional fine-tuning or target-domain data.

Introduction

In real-world applications, NMT models are frequently deployed in environments where the input domain is unknown or highly variable. While domain adaptation (tuning on specific target data) is effective, it is often impractical due to the lack of prior data for every possible niche. The challenge of Domain Generalization lies in retaining the specialized knowledge of known domains while learning to generalize to new ones. Traditional approaches often seek "domain-invariant" features, which can inadvertently strip away the stylistic and terminological nuances essential for high-quality translation. This work explores how to effectively mix domain-specific "flavors" to navigate unseen linguistic territories.

Methodology

The researchers propose a two-stage framework built upon a pre-trained encoder-decoder architecture (e.g., mBART).

1. Domain-Specific Parameter Learning

To capture domain-specific features efficiently, small trainable modules called Adapters are inserted into the Transformer layers. During this phase, the massive pre-trained backbone remains frozen, and only one adapter is trained per "seen" domain (e.g., Legal, Medical, IT). This parameter-efficient approach allows the model to specialize without catastrophic forgetting of its general knowledge.

2. AdapterFusion with LODO Training

To handle unseen domains, the model must learn to combine these specialized adapters. An AdapterFusion layer uses an attention mechanism to assign weights to each domain adapter based on the current input.

The AdapterFusion architecture within a Transformer layer, showing how Query, Key, and Value components interact to weight domain adapters.

3. The Leave-One-Domain-Out (LODO) Strategy

The core innovation is the LODO training strategy. Standard fusion training typically assumes access to the target domain. To simulate an "unseen" domain during training, the authors explicitly disable the matching adapter for a given batch. For instance, when processing "Medical" data, the Medical adapter is turned off, forcing the Fusion layer to represent Medical concepts by interpolating between other available adapters (e.g., Legal or IT). This prepares the model for inference on truly unknown domains where no matching adapter exists.

Experimental Setup

The method was evaluated on three language pairs: English-German (En-De), English-French (En-Fr), and English-Polish (En-Pl).

  • Source Domains (Seen): Legal (LAW), IT, Koran, Medical (MED), and Subtitles (SUB).
  • Unseen Test Domains: Literature (BOOK), presentation transcripts (TED), COVID-19 technical data (TICO19), and the ROBUST dataset.

Key Results

The LODO-Fusion method consistently outperformed strong baselines, including standard zero-shot evaluation and adversarial domain discriminators.

Comparative BLEU scores for different training algorithms across unseen domains. LODO(MD) consistently shows gains over zero-shot and discriminator-based baselines.
  • Performance Gains: The method achieved an average improvement of up to +0.8 BLEU score on unseen domains compared to zero-shot baselines.
  • Efficiency: The approach is highly parameter-efficient; while the mBART backbone contains 610 million parameters, only approximately 37 million parameters (adapters and fusion layers) are trained.
  • Base Model Impact: Interestingly, models pre-trained on generic data (WMT) often generalized better than those fine-tuned on multiple specific domains, suggesting that "generic" models retain a flexibility that is beneficial for fusion.

Discussion and Limitations

While effective, the LODO strategy relies on clearly labeled source domains to train the initial adapters. In scenarios where domain boundaries are blurry, the effectiveness of discrete adapters may diminish. Additionally, while parameter-efficient in storage, the use of multiple adapters and a fusion layer introduces a minor computational overhead during inference. Future research may explore scaling this approach to hundreds of domain adapters and refining the fusion mechanism for even more granular domain interpolation.

Summary updated

BibTeX
@inproceedings{vu2022domain,
  author = {Thuy-Trang Vu and Shahram Khadivi and Dinh Q. Phung and Gholamreza Haffari},
  title = {{Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.49},
}
Figure