Authors: Praveen Dakwale, Christof Monz (Informatics Institute, University of Amsterdam)
Introduction
Neural Machine Translation (NMT) systems are highly effective when trained on large-scale general-purpose datasets. However, their performance often falters in specialized domains where parallel data is scarce. The standard remedy, fine-tuning a pre-trained general model on domain-specific data, often leads to "catastrophic forgetting"—a significant drop in performance on the original general domain. This paper proposes two novel fine-tuning methods based on Knowledge Distillation (KD) that allow NMT models to adapt to new domains while preserving their general-domain translation capabilities without the computational costs associated with ensemble models.
Methodology
The authors introduce two approaches that leverage a "teacher" network (the original general-domain model) to guide a "student" network (the model being fine-tuned).
Multi-objective Fine-Tuning (MCL)
In the MCL approach, the loss function is modified to include a distillation component. During fine-tuning on in-domain data, the student model is trained to minimize both the standard cross-entropy loss against "hard targets" (the ground truth words) and the Kullback-Leibler (KL) divergence against "soft targets" provided by the teacher model. This secondary objective acts as a regularizer, ensuring the student's probability distribution does not deviate excessively from the teacher's general-domain knowledge.
Multi-output Layer Fine-Tuning (MLL)
The MLL approach treats the final output layer as task-specific. The architecture shares the encoder, decoder, and attention mechanisms between domains but introduces a new output layer for the target domain while retaining the original output layer for the general domain. During training, the shared parameters are updated using both the in-domain data and the distillation signal to maintain the functionality of the general-domain output layer.
Experimental Setup
The models were evaluated using the following datasets:
- General Domain (Out-of-Domain): WMT 2015 English-German (~4M sentences).
- In-Domain 1 (TED): IWSLT bitext (~170k sentences), stylistically distinct but sharing vocabulary with the general domain.
- In-Domain 2 (Medical): EMEA corpus (~200k sentences), representing a "distant" domain with highly specialized terminology.
Evaluation was conducted on newstest15 for the general domain and specific test sets for IWSLT and EMEA.
Results and Discussion
The proposed methods successfully mitigated catastrophic forgetting while achieving high in-domain performance.
- Domain Adaptation Performance: On the TED dataset, standard fine-tuning improved BLEU from 19.46 to 21.90 but degraded general domain scores from 18.54 to 12.68. In contrast, MCL achieved 22.19 on TED while maintaining a general domain score of 16.55.
- Distant Domain Robustness: The Medical domain test showed more extreme results. Standard fine-tuning caused the general domain score to collapse to 4.50. The MLL approach reached 22.33 on medical data while preserving a general domain score of 14.67.
- Inference Efficiency: Unlike ensemble methods, which require running multiple models simultaneously, MCL maintains a single-model architecture. Decoding speed for MCL was measured at 0.147 ms/sentence, nearly identical to standard fine-tuning (0.131 ms) and significantly faster than ensemble decoding (0.277 ms).
Conclusion
This work demonstrates that Knowledge Distillation is an effective tool for domain adaptation in NMT. By using soft targets as anchors and exploring task-specific parameters, models can learn specialized jargon without losing the underlying language structure learned from general data. Future research will focus on scaling these methods to iterative adaptation across multiple sequential domains and investigating the management of disjoint vocabularies.
Summary updated