DaLC: Domain Adaptation Learning Curve Prediction for Neural Machine Translation

Cheonbok Park, Hantae Kim, Ioan Calapodescu, Hyunchang Cho, Vassilina Nikoulina

Abstract

Domain Adaptation (DA) in Neural Machine Translation (NMT) typically involves fine-tuning a pre-trained general model on a sample of in-domain parallel data. However, without existing parallel data, practitioners lack a reliable method to estimate the potential benefits of DA or the volume of data required to reach specific quality targets. We propose Domain adaptation Learning Curve (DaLC), a framework that predicts prospective DA performance using only in-domain monolingual samples from the source language. By combining NMT encoder representations with instance- and corpus-level features, DaLC forecasts translation quality across various data scales. Our results demonstrate that instance-level analysis significantly outperforms traditional corpus-level frameworks, providing a robust tool for informed resource allocation in MT projects.

Introduction

In the field of Neural Machine Translation, Domain Adaptation is the primary method for specializing general-purpose models for niche fields like legal, medical, or technical translation. Despite its importance, DA presents a significant "chicken and egg" problem: to determine if adaptation is worthwhile, one needs a high-quality parallel dataset, but creating such a dataset is expensive and time-consuming.

Currently, MT practitioners often "fly blind," investing in data collection without knowing if the resulting model will meet quality requirements or if the gains will plateau quickly. DaLC addresses this by predicting the learning curve—the relationship between training data size and model performance—before a single sentence is translated by a human.

Methodology

DaLC is designed to predict translation quality (measured by chrF scores) at specific "anchor points," such as 1k, 10k, and 100k training samples. The framework moves beyond aggregate corpus statistics to analyze the nuances of individual sentences.

The DaLC Architecture

The model consists of two primary components:

  1. Encoder Pooling Component: This module extracts internal representations from a standard NMT encoder. The underlying theory is that the NMT model’s internal states already contain rich information regarding how "foreign" or difficult a new domain sentence is relative to its baseline training.
  2. Fusion Layer: This layer integrates the encoder data with several feature sets:
  3. Corpus-level features: Vocabulary overlap with the general model, type-token ratios, and dataset size.
  4. Instance difficulty features: Model-based uncertainty scores (entropy and margin scores) and cross-lingual similarity using LaBSE.

By training this predictor on historical adaptation data, DaLC learns to forecast the performance of a model on entirely unseen domains using only source-language text.

Experimental Setup and Results

The researchers evaluated DaLC across multiple language pairs (English-German, German-English, and English-Korean) using diverse datasets such as OPUS (Koran, IT, Medical, Law, Subtitles) and AI-Hub (Technology, Finance, Travel).

Key Findings

  • Superior Accuracy: DaLC significantly outperformed standard parametric baselines (like the $exp_3$ function) and XGBoost models. In German-English experiments, DaLC achieved an average Root Mean Square Error (RMSE) of 0.047, compared to 0.288 for the $exp_3$ baseline.
  • The "0k" Anchor Point: One of the most practical insights was that including the "0k" anchor point—the performance of the general model on the new domain without any adaptation—greatly stabilized predictions, providing a reliable baseline for the curve.
  • Efficiency: The predictor is remarkably lightweight, utilizing only about 4% of the parameters of a base Transformer model. This makes it a cost-effective tool to run prior to any major data collection effort.

Practical Implications for Budgeting

DaLC enables a shift from arbitrary data collection to budget optimization. The research demonstrates that distributing a data-creation budget equally across all domains is rarely the most efficient strategy. DaLC can reveal, for instance, that 10k samples in a "Medical" domain might yield a massive quality jump, while the same 10k in a "Subtitles" domain might offer diminishing returns. This allows practitioners to allocate resources where they will have the greatest impact on translation quality.

Limitations and Future Work

While DaLC provides a breakthrough in performance prediction, it currently relies solely on source-side information. This can lead to inaccuracies when the primary difficulty of a domain lies in target-language nuances, such as specific stylistic or formal requirements. Furthermore, the model's accuracy can decrease when faced with extreme domain shifts that fall far outside its training distribution.

Future research aims to integrate DaLC into Active Learning frameworks. Instead of merely predicting the curve, the model could help select the specific sentences that would be most beneficial to translate, allowing models to climb the learning curve as efficiently as possible.

Conclusion

DaLC offers a data-driven approach to one of the most persistent challenges in NMT. By providing accurate performance forecasts from monolingual data, it empowers organizations to make informed, strategic decisions about their translation workflows and data investments.

Summary updated

Abstract

Domain Adaptation (DA) of Neural Machine Translation (NMT) model often relies on a pre-trained general NMT model which is adapted to the new domain on a sample of in-domain parallel data. Without parallel data, there is no way to estimate the potential benefit of DA, nor the amount of parallel samples it would require. It is however a desirable functionality that could help MT practitioners to make an informed decision before investing resources in dataset creation. We propose a Domain adaptation Learning Curve prediction (DaLC) model that predicts prospective DA performance based on in-domain monolingual samples in the source language. Our model relies on the NMT encoder representations combined with various instance and corpus-level features. We demonstrate that instance-level is better able to distinguish between different domains compared to corpus-level frameworks proposed in previous studies. Finally, we perform in-depth analyses of the results highlighting the limitations of our approach, and provide directions for future research.
BibTeX
@inproceedings{park2022dalc,
  author = {Cheonbok Park and Hantae Kim and Ioan Calapodescu and Hyunchang Cho and Vassilina Nikoulina},
  title = {{DaLC: Domain Adaptation Learning Curve Prediction for Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=upNEMwaXh0},
}

Loading related content...

Figure