Learning Hidden Unit Contribution for Adapting Neural Machine Translation Models

David VilarNAACL 2018 · OpenReview

Abstract

Domain adaptation in Neural Machine Translation (NMT) often faces challenges such as catastrophic forgetting and high computational costs. This paper adapts Learning Hidden Unit Contribution (LHUC), a technique originally developed for speaker adaptation in speech recognition, to the NMT domain. By introducing a small set of learnable scaling parameters that modulate the output of hidden units, LHUC allows for efficient adaptation of pre-trained models to specific domains. Our results demonstrate translation quality improvements of up to 6.0 BLEU points with minimal parameter overhead and no degradation of the base model's performance.

Introduction

Training state-of-the-art NMT systems is a resource-intensive process that can take weeks. Adapting these "general-purpose" models to specialized domains (e.g., medical or technical texts) is typically achieved through fine-tuning. However, standard fine-tuning often leads to "catastrophic forgetting," where the model loses its ability to translate general-domain text as it over-specializes on the new data. Furthermore, updating all model parameters for every target domain is computationally expensive and requires storing a full copy of the model for every single target domain.

Methodology

The proposed method, LHUC, avoids modifying the original model parameters ($\theta$). Instead, it introduces a learnable vector $\rho$ for each layer. Each element in $\rho$ acts as a scaling factor for a specific hidden unit. The output of a layer is multiplied element-wise by a sigmoid-based scaling function:

$$a(x) = \frac{2}{1 + e^{-x}}$$

This function maps the parameters to a range between 0 and 2, where 0 effectively "mutes" a neuron, 1 leaves it unchanged, and 2 amplifies its contribution. This allows the model to re-weight existing features learned during general training to better suit the target domain.

LHUC modulates the contribution of individual hidden units by scaling their outputs.

The researchers evaluated this approach on German-to-English translation using the WMT dataset (~5M sentences) for out-of-domain (OOD) data and the IWSLT 2016 dataset (~197K sentences) for in-domain (ID) TED talk data.

Results and Discussion

LHUC demonstrates significant efficiency and performance benefits. In a "Growing Training" scenario, where a model trained on WMT is adapted to IWSLT, LHUC improved translation quality by up to 6.0 BLEU points over the non-adapted baseline.

BLEU scores comparing LHUC against standard fine-tuning (continuation) and baselines across different test sets.

Key findings include:

  • Performance: LHUC consistently outperformed standard fine-tuning (continuation of training). For example, on the TED 2013 test set, LHUC reached 37.7 BLEU compared to 36.8 for fine-tuning.
  • Stability: While fine-tuning caused a 1.7 BLEU point drop on out-of-domain data (catastrophic forgetting), LHUC maintained the original performance on the base domain when the scaling units were bypassed.
  • Efficiency: The adaptation adds only 12KB of overhead to a 335MB model and can be completed in less than an hour on a single K80 GPU.

Conclusion

LHUC provides a lightweight and effective mechanism for NMT domain adaptation. By re-weighting existing features rather than retraining the entire network, it achieves superior performance and stability compared to traditional fine-tuning. While this study focused on Recurrent Neural Networks (LSTMs), the principle of hidden unit modulation remains a promising direction for adapting modern Transformer-based models to diverse linguistic domains.

Summary updated

BibTeX
@inproceedings{vilar2018learning,
  author = {David Vilar},
  title = {{Learning Hidden Unit Contribution for Adapting Neural Machine Translation Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=Bkb_i7ZdbS},
}
Figure