Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation

Mitchell A. Gordon, Kevin Duh

Abstract

We explore best practices for training small, memory efficient machine translation models with sequence-level knowledge distillation in the domain adaptation setting. While both domain adaptation and knowledge distillation are widely-used, their interaction remains little understood. Our large-scale empirical results in machine translation (on three language pairs with three domains each) suggest distilling twice for best performance: once using general-domain data and again using in-domain data with an adapted teacher.

Introduction

Modern Neural Machine Translation (NMT) systems face a significant deployment challenge. While large-scale models achieve state-of-the-art performance, their size and computational requirements often preclude deployment on edge devices or mobile platforms. Furthermore, these models frequently struggle in specialized domains (e.g., legal, medical, or technical) where parallel training data is scarce.

Two common techniques address these issues: Domain Adaptation, which fine-tunes a general model on niche data, and Knowledge Distillation, which trains a compact "Student" model to mimic a larger "Teacher." However, the optimal way to combine these techniques remains an open question. This paper systematically investigates the interaction between distillation and adaptation to provide a robust training recipe for small, specialized NMT models.

Methodology: The Distill, Adapt, Distill Recipe

The core contribution of this work is a three-stage training process termed Distill, Adapt, Distill. This approach was identified through an empirical study of nine different training configurations, ranging from simple fine-tuning to complex multi-stage distillation.

Overview of the nine experimental configurations evaluated in the study. Configuration 9 represents the proposed "Distill, Adapt, Distill" recipe.

The winning strategy (Configuration 9) follows these steps:

  1. General-Domain Distillation: A small Student model is trained on a massive, general-domain dataset using sequence-level knowledge distillation from a large General-Domain Teacher.
  2. Teacher Adaptation: The large Teacher model is fine-tuned on the target in-domain data to become an "expert" in that specific field.
  3. In-Domain Distillation: The pre-distilled Student model is then fine-tuned on the in-domain data, using the adapted Teacher to provide the training signals (pseudo-labels).

Experimental Setup

The study evaluated three language pairs: German-English (De-En), Russian-English (Ru-En), and Chinese-English (Zh-En). Models were tested across two distinct specialized domains:

  • WIPO: Technical patent abstracts from the World Intellectual Property Organization.
  • TED: Informal, conversational transcripts of TED talks.

Model architectures ranged from "Large" (12 layers) used for teachers, down to "Tiny" (2 layers) for highly efficient students.

Results and Analysis

The experiments yielded several key insights into the behavior of small models in specialized domains.

Comparison of General-Domain vs. In-Domain BLEU scores across different model sizes (Tiny, Small, Medium) and training roles (Teacher, Student, Baseline).

Distillation and Adaptability

A critical finding is that distillation does not hinder a model's ability to adapt. Distilled students remained highly flexible and benefited significantly from subsequent in-domain fine-tuning. In fact, distillation often provided a superior starting point compared to training on raw data alone.

The Importance of Teacher Expertise

The performance of the student is heavily dependent on the quality of the teacher's signals in the target domain. Using an adapted teacher for the second stage of distillation consistently outperformed using a general-purpose teacher. For specialized domains like WIPO patents, having a "domain expert" teacher is essential for transferring nuanced terminology and style to the student.

Efficiency Gains for Small Models

The "Tiny" (2-layer) models saw the most dramatic relative improvements from the Distill, Adapt, Distill recipe. While larger "Medium" models sometimes achieved high performance without distillation, the smallest models required the inductive bias provided by general-domain distillation to learn basic translation rules before specializing.

Conclusion

The "Distill, Adapt, Distill" recipe provides a clear path for creating high-performance, compact NMT models for specialized domains. While this multi-stage process increases training time and computational cost (up to 4x compared to standard training), the resulting gains in BLEU scores and deployment efficiency make it a compelling choice for real-world applications on resource-constrained hardware.

Summary updated

Abstract

We explore best practices for training small, memory efficient machine translation models with sequence-level knowledge distillation in the domain adaptation setting. While both domain adaptation and knowledge distillation are widely-used, their interaction remains little understood. Our large-scale empirical results in machine translation (on three language pairs with three domains each) suggest distilling twice for best performance: once using general-domain data and again using in-domain data with an adapted teacher.
BibTeX
@inproceedings{gordon2020distill,
  author = {Mitchell A. Gordon and Kevin Duh},
  title = {{Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation}},
  booktitle = {NGT @ ACL},
  year = {2020},
  url = {https://openreview.net/forum?id=7biQ3eh2pCH},
}
Figure