Incremental Domain Adaptation for Neural Machine Translation in Low-Resource Settings

Marimuthu Kalimuthu, Michael Barz, Daniel Sonntag

Abstract

Adapting Neural Machine Translation (NMT) models to specialized domains is often hindered by the high cost of acquiring human-translated data. This paper presents a practical approach to incremental domain adaptation using an advanced query strategy based on sentence embeddings. By identifying and selecting the most "informative" sentences from a new domain—specifically those that are semantically distant from the generic training data—the proposed method significantly reduces the human labor required for fine-tuning. Experiments in Arabic-to-English translation demonstrate that this strategy can reduce human translation costs by over 50% without sacrificing translation quality, outperforming standard random sampling baselines.

Introduction

Neural Machine Translation (NMT) has achieved state-of-the-art performance but remains highly dependent on large-scale parallel corpora. When applied to specialized domains (e.g., climate change, legal documents), generic models often fail to capture domain-specific nuances. Domain adaptation through fine-tuning is the standard remedy, yet the creation of high-quality parallel data for new domains is expensive and time-consuming. This research addresses the challenge of sample efficiency: identifying which sentences, if translated by a human, provide the greatest benefit to the model's adaptation.

Methodology

The proposed framework focuses on an active learning-inspired sampling strategy that operates on monolingual data from the target domain.

1. Generic Model and Embeddings

The researchers established a baseline generic model ($M_{gen}$) using an 8-layered bidirectional RNN trained on the LDC Newswire corpus (1.3 million Arabic-English pairs). To represent sentences numerically, they developed AraSIF, an adaptation of the Smooth Inverse Frequency (SIF) method for Arabic, generating 300-dimensional semantic vectors.

2. Advanced Query Strategy

Instead of random sampling, the "Advanced Query Strategy" filters the new domain corpus ($S_{src}^{new}$) to select the most informative samples for human translation. The process involves:

  • Calculating a reference point ($e_{ref}$) representing the mean embedding of the generic training data.
  • Measuring the cosine distance between new, untranslated domain sentences and $e_{ref}$.
  • Selecting sentences from the "long-tail" of the distribution—those most semantically distant or underrepresented in the generic data.
The proposed incremental domain adaptation framework, highlighting the filtering process (f) that reduces human labor.

The Long-Tail Hypothesis

The core insight is that sentences falling outside the typical distribution of the generic corpus (e.g., outside the 25th and 75th percentiles) force the model to adapt its weights more significantly. These "outlier" sentences represent knowledge the model lacks, making them more valuable for fine-tuning than sentences similar to what the model has already seen.

Distribution of cosine similarities between sentence embeddings and the generic reference point across different datasets.

Experimental Results

The strategy was evaluated on two target domains: MEDAR (Climate Change) and GlobalVoices (Politics and Human Rights).

  • Labor Reduction: For the MEDAR dataset, the advanced strategy achieved a 66.25% reduction in human labor, requiring only 135 samples compared to the 400 needed for random sampling to reach similar performance.
  • Translation Quality: Despite the smaller sample size, the model fine-tuned on "advanced" samples achieved a BLEU score of 19.34, significantly higher than the generic baseline (18.6).
  • Consistency: On GlobalVoices, the method excluded nearly 58% of the data while maintaining translation quality comparable to the full random baseline.

Conclusion and Future Work

This research demonstrates that monolingual filtering using sentence embeddings is a highly effective way to prioritize human translation efforts. By focusing on the most informative samples, researchers can achieve superior domain adaptation at a fraction of the cost. Future work will explore clustering-based approaches to compare sentences against multiple topic clusters rather than a single mean, and the integration of human-in-the-loop post-editing systems.

Summary updated

BibTeX
@inproceedings{kalimuthu2019incremental,
  author = {Marimuthu Kalimuthu and Michael Barz and Daniel Sonntag},
  title = {{Incremental Domain Adaptation for Neural Machine Translation in Low-Resource Settings}},
  booktitle = {WANLP @ ACL},
  year = {2019},
  doi = {10.18653/v1/w19-4601},
}
Figure