Vocabulary Adaptation for Domain Adaptation in Neural Machine Translation

Shoetsu Sato, Jin Sakuma, Naoki Yoshinaga, Masashi Toyoda, Masaru KitsuregawaEMNLP 2020 · DOI

Abstract

Neural Machine Translation (NMT) models often suffer from performance degradation when applied to specialized domains different from their training data. This "domain gap" is primarily driven by unknown technical terms and semantic shifts in common vocabulary. This paper introduces Vocabulary Adaptation, a novel method that projects target-domain word embeddings into a pre-trained model’s embedding space prior to fine-tuning. By aligning these representations, the authors achieve significant gains of up to 3.86 BLEU points over standard fine-tuning methods.

The Challenge of Domain Gaps

When transitioning an NMT model from a resource-rich general domain (e.g., movie subtitles) to a specialized one (e.g., scientific abstracts), the model encounters two major obstacles:

  1. Unknown Words: Technical terminology (like "angstrom") absent from the source training data.
  2. Semantic Shift: Words that change meaning across contexts. For instance, "conductor" may refer to an orchestra leader in general text but a physical material in scientific literature.

Standard fine-tuning often fails to resolve these issues because the model remains anchored to its original source-domain embedding layers, leading to poor handling of new meanings or nonsensical subword fragmentation.

Proposed Methodology

The authors propose a three-step pipeline designed to adapt the model's vocabulary before the final fine-tuning phase:

  1. Induce Target Embeddings: Word embeddings are trained on large-scale target-domain monolingual data using Continuous Bag-of-Words (CBOW).
  2. Cross-Domain Projection: The newly induced target embeddings are mapped into the pre-trained model's existing embedding space. The authors highlight Locally Linear Mapping (LLM) as a superior technique to standard linear transformations, as it preserves the local relationships between words even when the global structures of the two spaces differ.
  3. Fine-tuning: The original embedding layers are replaced with the projected target embeddings, and the entire NMT model is fine-tuned on a small set of target-domain parallel data.
The Vocabulary Adaptation pipeline, illustrating the flow from pre-training on massive source data to fine-tuning on small target data via cross-domain embedding projection.

Key Insights

  • Subwords are Insufficient: While subword tokenization (like BPE) prevents "out-of-vocabulary" errors, it can exacerbate semantic shift by forcing target-domain meanings onto inappropriate source-domain fragments.
  • Local Topology is Crucial: Global linear projections often fail because the "shape" of embedding spaces varies between tasks. LLM succeeds by focusing on local neighborhoods—keeping "conductor" near "electricity" rather than attempting a perfect global alignment.
Conceptual illustration of the linear transformation process used to align target domain space with source domain space to create a unified cross-domain embedding space.

Experimental Results

The method was evaluated on English→Japanese (Subtitles to Scientific Abstracts) and German→English (IT to Law) tasks:

  • Performance Gains: The VA-LLM approach outperformed standard fine-tuning by 3.86 BLEU points in En→Ja and 3.28 points in De→En.
  • Efficiency: Adapted models converged significantly faster, reaching superior performance levels in a fraction of the training updates required by baseline methods.
  • Qualitative Improvement: The model successfully translated complex technical terms like "spinal stenosis" that the baseline model previously ignored or corrupted.

Conclusion

Vocabulary Adaptation demonstrates that modifying a model's internal representations is as vital as augmenting training data. While it requires target-domain monolingual data, the resulting gains in translation accuracy and training efficiency make it a powerful tool for specialized NMT applications.

Summary updated

BibTeX
@inproceedings{sato2020vocabulary,
  author = {Shoetsu Sato and Jin Sakuma and Naoki Yoshinaga and Masashi Toyoda and Masaru Kitsuregawa},
  title = {{Vocabulary Adaptation for Domain Adaptation in Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.381},
}
Figure