Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation

Mitchell A. Gordon, Kevin Duh

Abstract

We explore best practices for training small, memory efficient machine translation models with sequence-level knowledge distillation in the domain adaptation setting. While both domain adaptation and knowledge distillation are widely-used, their interaction remains little understood. Our large-scale empirical results in machine translation (on three language pairs with three domains each) suggest distilling twice for best performance: once using general-domain data and again using in-domain data with an adapted teacher.

Introduction

Modern Neural Machine Translation (NMT) systems, typically based on the Transformer architecture, achieve state-of-the-art accuracy but at a significant computational cost. These "Large" models are often too slow for real-time inference and too memory-intensive for edge devices. Furthermore, they frequently struggle with specialized domains—such as legal, medical, or technical patent text—where training data is scarce.

To address these challenges, researchers typically employ two techniques: Domain Adaptation (fine-tuning a general model on niche data) and Knowledge Distillation (training a small "Student" model to mimic a large "Teacher"). However, the optimal way to combine these methods has remained unclear. This work systematically explores the interaction between distillation and adaptation, proposing a high-performance three-stage pipeline: "Distill, Adapt, Distill."

The Experimental Landscape

The researchers performed an extensive empirical study, testing nine different configurations for combining distillation and adaptation. These configurations varied based on how the student model was initialized (randomly, from a general-domain baseline, or from a general-domain student) and which teacher model provided the distillation signal (none, a general-domain teacher, or an adapted in-domain teacher).

The nine experimental configurations for combining knowledge distillation and domain adaptation. The winning "Distill, Adapt, Distill" approach corresponds to Configuration 9.

The study spanned three language pairs—German-English (De-En), Russian-English (Ru-En), and Chinese-English (Zh-En)—and two distinct niche domains: WIPO (technical patent abstracts) and TED (conversational transcripts).

The "Distill, Adapt, Distill" Recipe

The most effective approach, identified as Configuration 9, follows a three-stage recipe:

  1. Distill General Knowledge: Train a large teacher on general-domain data, then use it to distill that knowledge into a small student model.
  2. Adapt the Student: Fine-tune that small student model directly on the target in-domain data.
  3. Distill In-Domain Knowledge: Take a large teacher that has been adapted to the domain and use it to distill the in-domain data one more time into the student.

This method ensures the student captures the fundamental rules of language from the general domain before specializing in the nuances of the target domain. Interestingly, the researchers found that distillation does not "break" adaptability; distilled models fine-tune to new domains just as well as, or better than, standard baselines.

Key Results and Insights

The experiments yielded several critical findings for NMT practitioners:

  • Significant BLEU Gains: Moving from a general-domain model to an in-domain model typically yields a jump of 5 to 10 BLEU points. The second stage of distillation (using an adapted teacher) provides an additional 0.1 to 2.0 BLEU point boost over simple adaptation.
  • Model Size and Efficiency: While larger models perform best, the "Distill, Adapt, Distill" pipeline allows "Small" and "Tiny" models to punch well above their weight class. Medium-sized models (6 layers, 512 hidden) were found to be "big enough" to learn patterns directly, seeing fewer gains from distillation than their smaller counterparts.
General Domain vs. In-Domain BLEU scores across model sizes. The plot illustrates that while larger models (T) perform best, students (S) trained with the proposed pipeline closely track their performance even at smaller scales.
  • The Importance of Direct Exposure: Students must be exposed to general-domain data (either via pre-training or distillation) before domain-specific distillation. Attempting to learn general language rules solely through an adapted teacher is significantly less effective.

Conclusion

The "Distill, Adapt, Distill" workflow provides a robust recipe for deploying high-quality NMT on resource-constrained devices. By leveraging an adapted teacher for a final round of in-domain distillation, researchers can produce "Tiny" models that remain competitive with much larger baselines. While the training process is computationally intensive, the resulting efficiency gains during inference make it a compelling strategy for production environments.

Summary updated

Abstract

We explore best practices for training small, memory efficient machine translation models with sequence-level knowledge distillation in the domain adaptation setting. While both domain adaptation and knowledge distillation are widely-used, their interaction remains little understood. Our large-scale empirical results in machine translation (on three language pairs with three domains each) suggest distilling twice for best performance: once using general-domain data and again using in-domain data with an adapted teacher.
BibTeX
@inproceedings{gordon2020distill,
  author = {Mitchell A. Gordon and Kevin Duh},
  title = {{Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation}},
  booktitle = {NGT @ ACL},
  year = {2020},
  url = {https://openreview.net/forum?id=7biQ3eh2pCH},
}
Figure