Abstract
Many multi-domain neural machine translation (NMT) models achieve knowledge transfer by enforcing one encoder to learn shared embedding across domains. However, this design lacks adaptation to individual domains. To overcome this limitation, we propose a novel multi-domain NMT model using individual modules for each domain, on which we apply word-level, adaptive and layer-wise domain mixing. We first observe that words in a sentence are often related to multiple domains. Hence, we assume each word has a domain proportion, which indicates its domain preference. Then word representations are obtained by mixing their embedding in individual domains based on their domain proportions. We show this can be achieved by carefully designing multi-head dot-product attention modules for different domains, and eventually taking weighted averages of their parameters by word-level layer-wise domain proportions. Through this, we can achieve effective domain knowledge sharing, and capture fine-grained domain-specific knowledge as well. Our experiments show that our proposed model outperforms existing ones in several NMT tasks.
Introduction
In Neural Machine Translation (NMT), a persistent challenge is balancing the benefits of large-scale "unified" models with the precision of domain-specific specialized models. Unified models, trained on diverse datasets, benefit from cross-domain knowledge transfer but often fail to capture the nuances of specific fields like law, medicine, or technical news. Conversely, specialized models excel in their niche but suffer from data scarcity.
Existing solutions like instance weighting or domain-aware embeddings often rely on a single shared encoder. This architecture lacks the flexibility to handle "domain heterogeneity" at a granular level. For instance, the word "article" carries distinct semantic weights in legal versus journalistic contexts. This paper introduces a Transformer-based approach that maintains separate parameters for different domains and blends them dynamically for every word, allowing for a more nuanced adaptation.
Methodology: Word-Level Adaptive Layer-wise Domain Mixing
The proposed model moves beyond static domain tags by introducing Word-Level Adaptive Layer-wise Domain Mixing. The architecture maintains $k$ sets of parameters—one for each target domain—and dynamically interpolates them based on the context of each word.
Domain Proportion and Parameter Mixing
For every word embedding, the model employs a smoothed softmax layer to predict a "domain proportion" ($D(x)$). This proportion represents a probability distribution across the $k$ possible domains. Instead of selecting a single domain, the model calculates a weighted average of the attention parameters (the $W_Q, W_K, W_V$ matrices) based on these proportions.
Layer-wise Adaptation
This mixing is not a one-time operation at the input layer. Instead, domain proportions are recalculated at every layer of the Transformer encoder and decoder. This allows the model to adapt its internal representations as it processes deeper semantic features.
The model is trained using a composite loss function that combines the standard translation loss ($L_{gen}$) with a domain classification loss ($L_{mix}$), which encourages the model to correctly identify the domain context of a word based on its surroundings.
Experimental Setup
The model was evaluated across three language pairs with varying domain characteristics:
- English-to-German (EN-DE): News (Europarl) and TED talks (IWLST).
- English-to-French (EN-FR): TED talks and Medical data (Medline).
- Chinese-to-English (ZH-EN): A complex four-domain setup including Laws, News, Speech, and Thesis data from the UM-Corpus and LDC.
Results and Discussion
The proposed model consistently outperformed strong baselines, including standard Multitask Learning (MTL) and Adversarial Learning (AdvL) approaches.
Performance Gains
In the EN-DE task, the model achieved a BLEU score of 27.78 on News (compared to 26.06 for the unified baseline) and 30.30 on TED (compared to 28.11). The gains were even more pronounced in the EN-FR task, reaching 40.60 on TED and 54.39 on Medical data. These results suggest that the specialized modules effectively capture domain-specific knowledge without losing the benefits of shared training.
Emergent Linguistic Patterns
A key insight from the analysis is the "general-to-specific" behavior of the layers. Domain proportions tend to be more uniform in the lower layers, where the model learns general syntax and grammar. In the higher layers, the proportions become more skewed toward specific domains, indicating that specialized knowledge is integrated later in the encoding process. This allows the model to resolve word ambiguities effectively—for example, correctly identifying whether "article" refers to a legal clause or a news story based on the surrounding context.
Conclusion
By introducing word-level, layer-wise domain mixing, this research provides a flexible framework for multi-domain NMT. The model successfully balances the need for shared linguistic knowledge with the requirement for domain-specific expertise. While challenges remain in extremely difficult domains like academic theses, the approach offers a significant step forward in fine-grained domain adaptation.
Summary updated