Abstract
Researchers from Georgia Tech and ByteDance have introduced a novel Multi-Domain Neural Machine Translation (NMT) framework that moves away from the traditional "one-size-fits-all" encoder. By utilizing individual modules for different domains and blending them at both the word and layer levels, the model achieves fine-grained adaptation. This approach allows the system to translate polysemous words correctly based on specific context without sacrificing performance on other domains. Experimental results across multiple language pairs demonstrate that this adaptive mixing consistently outperforms standard multitask and adversarial learning baselines.
Introduction
Standard NMT models thrive on massive datasets but often struggle when applied to specific domains like medical, legal, or technical fields. Traditionally, researchers have used two main approaches: unified models that pool all data together, and fine-tuning. Unified models often result in a "jack of all trades, master of none" scenario where domain-specific nuances are lost. Conversely, fine-tuning a general model on a specific domain usually leads to "catastrophic forgetting," where the model loses its ability to translate other domains effectively.
The core challenge lies in the fact that words are often polysemous across domains. For example, the word "article" may refer to a legal clause in a law domain but a piece of writing in a media domain. Existing models using a single shared encoder struggle to capture these subtle, word-level domain shifts. This paper addresses this by proposing a mechanism that adapts at the token level within each layer of the network.
Methodology
The proposed approach is centered on Word-Level Adaptive Layer-wise Domain Mixing. Instead of a single shared encoder, the architecture incorporates individual multi-head attention modules for each target domain.
Domain Proportions and Soft Mixing
For every word in a sentence, the model calculates a "domain proportion" ($D(x)$), which acts as a set of weights indicating the word's relevance to each specific domain (e.g., 70% Law, 30% News). Rather than selecting a single domain, the model performs "soft-mixing" by taking a weighted average of the parameters from the domain-specific modules. This happens at the word level, allowing every token in a sentence to have a unique blend of domain knowledge.
Layer-wise Adaptation
This mixing is dynamic rather than static. As a word progresses from the bottom layers of the Transformer (handling basic syntax) to the top layers (processing high-level semantics), the domain proportions are re-calculated. This allows the model to refine its domain assignment as it gains more contextual information.
Training Objective
The model is optimized using a composite loss function: a standard translation loss ($L_{gen}$) combined with a domain classification loss ($L_{mix}$). The latter encourages the model to correctly identify the domain of each word, reinforcing the specialization of the underlying modules.
Experimental Results
The model was evaluated on three diverse language pairs and domain sets: English-to-German (News and TED), English-to-French (TED and Medical), and Chinese-to-English (Laws, News, Speech, and Thesis).
As shown in the training curves, the proposed mixing approach achieves significantly lower perplexity and faster convergence compared to standard baselines like Multitask Learning (MTL) and Adversarial Learning (AdvL).
Key performance highlights include:
- EN-DE: Achieved BLEU scores of 27.78 (News) and 30.30 (TED), outperforming embedding-based baselines by up to 0.9 points.
- EN-FR: Showed significant gains in the TED domain (40.60 BLEU vs. 39.79 for the best baseline).
- ZH-EN: In a complex four-domain task, the model outperformed the "Mixed" data baseline by nearly 2 BLEU points in the Law domain, demonstrating its ability to handle high heterogeneity.
Discussion and Insights
Analysis of the model's behavior revealed several key insights:
- Context-Dependent Identity: A word’s domain identity is not fixed. At lower layers, domain proportions are diverse and often related to word frequency. At higher layers, proportions become more specialized (skewed), indicating the model commits to a domain as context becomes clearer.
- Knowledge Sharing vs. Specialization: Simple functional words (e.g., "and", "the") tend to have uniform domain proportions, effectively sharing knowledge across all modules. In contrast, domain-heavy words (e.g., "phenomenon") maintain strong ties to specific domain modules throughout the network.
- Disambiguation: By allowing different positions in a sentence to carry different domain information, the model successfully disambiguates words like "article," treating them differently based on their specific placement and surrounding context.
Conclusion
While the model shows great promise, it faces challenges in extremely data-sparse domains like "Thesis" in the Chinese-to-English task. Additionally, as the number of domains increases, parameter scaling becomes a concern. Future work may involve more efficient parameter-sharing techniques, such as Adapters, to scale the system to dozens of domains. The authors also suggest that word-level mixing could be extended to other multi-domain NLP tasks, including sentiment analysis and named entity recognition.
Summary updated