Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing

Haoming Jiang, Chen Liang, Chong Wang, Tuo Zhao

Abstract

Researchers from Georgia Tech and ByteDance have introduced a novel Multi-Domain Neural Machine Translation (NMT) framework that moves away from the traditional "one-size-fits-all" encoder. By utilizing individual modules for different domains and blending them at both the word and layer levels, the model achieves fine-grained adaptation. This approach allows the system to translate polysemous words correctly based on specific context without sacrificing performance on other domains. Experimental results across multiple language pairs demonstrate that this adaptive mixing consistently outperforms standard multitask and adversarial learning baselines.

Introduction

Standard NMT models thrive on massive datasets but often struggle when applied to specific domains like medical, legal, or technical fields. Traditionally, researchers have used two main approaches: unified models that pool all data together, and fine-tuning. Unified models often result in a "jack of all trades, master of none" scenario where domain-specific nuances are lost. Conversely, fine-tuning a general model on a specific domain usually leads to "catastrophic forgetting," where the model loses its ability to translate other domains effectively.

The core challenge lies in the fact that words are often polysemous across domains. For example, the word "article" may refer to a legal clause in a law domain but a piece of writing in a media domain. Existing models using a single shared encoder struggle to capture these subtle, word-level domain shifts. This paper addresses this by proposing a mechanism that adapts at the token level within each layer of the network.

Methodology

The proposed approach is centered on Word-Level Adaptive Layer-wise Domain Mixing. Instead of a single shared encoder, the architecture incorporates individual multi-head attention modules for each target domain.

Domain Proportions and Soft Mixing

For every word in a sentence, the model calculates a "domain proportion" ($D(x)$), which acts as a set of weights indicating the word's relevance to each specific domain (e.g., 70% Law, 30% News). Rather than selecting a single domain, the model performs "soft-mixing" by taking a weighted average of the parameters from the domain-specific modules. This happens at the word level, allowing every token in a sentence to have a unique blend of domain knowledge.

Layer-wise Adaptation

This mixing is dynamic rather than static. As a word progresses from the bottom layers of the Transformer (handling basic syntax) to the top layers (processing high-level semantics), the domain proportions are re-calculated. This allows the model to refine its domain assignment as it gains more contextual information.

The proposed architecture featuring domain mixing in both the encoder and decoder layers.

Training Objective

The model is optimized using a composite loss function: a standard translation loss ($L_{gen}$) combined with a domain classification loss ($L_{mix}$). The latter encourages the model to correctly identify the domain of each word, reinforcing the specialization of the underlying modules.

Experimental Results

The model was evaluated on three diverse language pairs and domain sets: English-to-German (News and TED), English-to-French (TED and Medical), and Chinese-to-English (Laws, News, Speech, and Thesis).

As shown in the training curves, the proposed mixing approach achieves significantly lower perplexity and faster convergence compared to standard baselines like Multitask Learning (MTL) and Adversarial Learning (AdvL).

Perplexity comparison across different training epochs for various domain adaptation methods.

Key performance highlights include:

  • EN-DE: Achieved BLEU scores of 27.78 (News) and 30.30 (TED), outperforming embedding-based baselines by up to 0.9 points.
  • EN-FR: Showed significant gains in the TED domain (40.60 BLEU vs. 39.79 for the best baseline).
  • ZH-EN: In a complex four-domain task, the model outperformed the "Mixed" data baseline by nearly 2 BLEU points in the Law domain, demonstrating its ability to handle high heterogeneity.

Discussion and Insights

Analysis of the model's behavior revealed several key insights:

  • Context-Dependent Identity: A word’s domain identity is not fixed. At lower layers, domain proportions are diverse and often related to word frequency. At higher layers, proportions become more specialized (skewed), indicating the model commits to a domain as context becomes clearer.
  • Knowledge Sharing vs. Specialization: Simple functional words (e.g., "and", "the") tend to have uniform domain proportions, effectively sharing knowledge across all modules. In contrast, domain-heavy words (e.g., "phenomenon") maintain strong ties to specific domain modules throughout the network.
  • Disambiguation: By allowing different positions in a sentence to carry different domain information, the model successfully disambiguates words like "article," treating them differently based on their specific placement and surrounding context.

Conclusion

While the model shows great promise, it faces challenges in extremely data-sparse domains like "Thesis" in the Chinese-to-English task. Additionally, as the number of domains increases, parameter scaling becomes a concern. Future work may involve more efficient parameter-sharing techniques, such as Adapters, to scale the system to dozens of domains. The authors also suggest that word-level mixing could be extended to other multi-domain NLP tasks, including sentiment analysis and named entity recognition.

Summary updated

Abstract

Many multi-domain neural machine translation (NMT) models achieve knowledge transfer by enforcing one encoder to learn shared embedding across domains. However, this design lacks adaptation to individual domains. To overcome this limitation, we propose a novel multi-domain NMT model using individual modules for each domain, on which we apply word-level, adaptive and layer-wise domain mixing. We first observe that words in a sentence are often related to multiple domains. Hence, we assume each word has a domain proportion, which indicates its domain preference. Then word representations are obtained by mixing their embedding in individual domains based on their domain proportions. We show this can be achieved by carefully designing multi-head dot-product attention modules for different domains, and eventually taking weighted averages of their parameters by word-level layer-wise domain proportions. Through this, we can achieve effective domain knowledge sharing, and capture fine-grained domain-specific knowledge as well. Our experiments show that our proposed model outperforms existing ones in several NMT tasks.
BibTeX
@inproceedings{jiang2020multi,
  author = {Haoming Jiang and Chen Liang and Chong Wang and Tuo Zhao},
  title = {{Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=P7lVJufHWa},
}
Figure