Fine-Tuning for Neural Machine Translation with Limited Degradation across In- and Out-of-Domain Data

Praveen Dakwale, Christof MonzMTSummit 2017 · OpenReview

Authors: Praveen Dakwale, Christof Monz (Informatics Institute, University of Amsterdam)

Introduction

Neural Machine Translation (NMT) systems are highly effective when trained on large-scale general-purpose datasets. However, their performance often falters in specialized domains where parallel data is scarce. The standard remedy, fine-tuning a pre-trained general model on domain-specific data, often leads to "catastrophic forgetting"—a significant drop in performance on the original general domain. This paper proposes two novel fine-tuning methods based on Knowledge Distillation (KD) that allow NMT models to adapt to new domains while preserving their general-domain translation capabilities without the computational costs associated with ensemble models.

Methodology

The authors introduce two approaches that leverage a "teacher" network (the original general-domain model) to guide a "student" network (the model being fine-tuned).

Multi-objective Fine-Tuning (MCL)

In the MCL approach, the loss function is modified to include a distillation component. During fine-tuning on in-domain data, the student model is trained to minimize both the standard cross-entropy loss against "hard targets" (the ground truth words) and the Kullback-Leibler (KL) divergence against "soft targets" provided by the teacher model. This secondary objective acts as a regularizer, ensuring the student's probability distribution does not deviate excessively from the teacher's general-domain knowledge.

Architecture of Multi-objective Fine-Tuning (MCL) using Knowledge Distillation.

Multi-output Layer Fine-Tuning (MLL)

The MLL approach treats the final output layer as task-specific. The architecture shares the encoder, decoder, and attention mechanisms between domains but introduces a new output layer for the target domain while retaining the original output layer for the general domain. During training, the shared parameters are updated using both the in-domain data and the distillation signal to maintain the functionality of the general-domain output layer.

Architecture of Multi-output Layer Fine-Tuning (MLL) with task-specific output layers.

Experimental Setup

The models were evaluated using the following datasets:

  • General Domain (Out-of-Domain): WMT 2015 English-German (~4M sentences).
  • In-Domain 1 (TED): IWSLT bitext (~170k sentences), stylistically distinct but sharing vocabulary with the general domain.
  • In-Domain 2 (Medical): EMEA corpus (~200k sentences), representing a "distant" domain with highly specialized terminology.

Evaluation was conducted on newstest15 for the general domain and specific test sets for IWSLT and EMEA.

Results and Discussion

The proposed methods successfully mitigated catastrophic forgetting while achieving high in-domain performance.

  • Domain Adaptation Performance: On the TED dataset, standard fine-tuning improved BLEU from 19.46 to 21.90 but degraded general domain scores from 18.54 to 12.68. In contrast, MCL achieved 22.19 on TED while maintaining a general domain score of 16.55.
  • Distant Domain Robustness: The Medical domain test showed more extreme results. Standard fine-tuning caused the general domain score to collapse to 4.50. The MLL approach reached 22.33 on medical data while preserving a general domain score of 14.67.
  • Inference Efficiency: Unlike ensemble methods, which require running multiple models simultaneously, MCL maintains a single-model architecture. Decoding speed for MCL was measured at 0.147 ms/sentence, nearly identical to standard fine-tuning (0.131 ms) and significantly faster than ensemble decoding (0.277 ms).

Conclusion

This work demonstrates that Knowledge Distillation is an effective tool for domain adaptation in NMT. By using soft targets as anchors and exploring task-specific parameters, models can learn specialized jargon without losing the underlying language structure learned from general data. Future research will focus on scaling these methods to iterative adaptation across multiple sequential domains and investigating the management of disjoint vocabularies.

Summary updated

BibTeX
@inproceedings{dakwale2017fine,
  author = {Praveen Dakwale and Christof Monz},
  title = {{Fine-Tuning for Neural Machine Translation with Limited Degradation across In- and Out-of-Domain Data}},
  booktitle = {MTSummit},
  year = {2017},
  url = {https://openreview.net/forum?id=3ea4_o_ypd},
}
Figure