Curriculum Learning for Domain Adaptation in Neural Machine Translation

Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, Kevin Duh

Introduction

Neural Machine Translation (NMT) models have achieved remarkable success but remain notoriously "brittle." While they perform exceptionally well on data similar to their training set, performance often collapses when faced with a new domain—for instance, a model trained on news text struggling with legal patents.

The standard approach to this issue is Domain Adaptation, typically through "continued training" or fine-tuning. In this process, a general-purpose model is further trained on a small set of in-domain data. To enhance this, researchers often supplement the training with "pseudo" in-domain sentences selected from massive, noisy web crawls like Paracrawl. However, a significant challenge remains: how much extra data should be included? Too little data misses helpful vocabulary, while too much introduces noise that degrades performance.

Overview of the domain adaptation process using continued training on a mix of in-domain and unlabeled domain data.

Probabilistic Curriculum Learning

To address the data selection trade-off, Zhang et al. propose a Probabilistic Curriculum Learning (CL) strategy. Instead of treating all selected data as equally relevant, they structure the training process like a school syllabus, introducing data in a "most-similar-to-least-similar" schedule.

The method involves four key steps:

  1. Similarity Scoring: Sentences from a general web crawl are ranked based on their similarity to the target domain using methods like Moore-Lewis (cross-entropy differences) and Cynical Data Selection.
  2. Sharding: The ranked sentences are grouped into "shards."
  3. Phased Training: Training is divided into phases. Initially, the model only sees the most similar shard. In subsequent phases, less similar shards are gradually added to the training pool.
  4. Probabilistic Sampling: To maintain the benefits of randomization essential for deep learning, the approach shuffles data within the current pool and employs standard practices like bucketing by sentence length.

Experimental Setup

The researchers evaluated their approach across four adaptation tasks involving two language pairs: German-English (de-en) and Russian-English (ru-en). The target domains were TED Talk transcripts and Patents from the World International Property Organization. General domain data comprised approximately 28 million pairs for German and 51 million for Russian, while Paracrawl provided the source for unlabeled web data.

Key Results and Insights

The curriculum approach consistently outperformed standard continued training across all domains and language pairs, achieving gains of up to 3.22 BLEU points over the adapted baseline.

BLEU scores across different tasks as the number of selected Paracrawl sentences increases. Curriculum Learning (CL) methods (solid lines) maintain high performance even as more noisy data is added, whereas standard (std) methods (dashed lines) often decline.

Robustness to Noise

A critical finding was the model's robustness to noise. In standard continued training, performance often peaks and then declines as more web-crawled data is added because the "tail" of the data is of lower quality. In contrast, the CL models maintained high performance even when adding up to 4 million extra sentences, effectively buffering against the negative impact of less relevant data.

Lexical Choice and Error Analysis

Using $S^4$ error analysis, the authors discovered that the curriculum does more than just help the model learn new vocabulary (SEEN errors). Its primary benefit lies in helping the model choose the correct meaning for words it already knows but which have different nuances in the new domain (SCORE and SENSE errors). This suggests that the structured introduction of data helps the model refine its internal representations for domain-specific contexts.

Conclusion

Curriculum learning provides a robust framework for domain adaptation in NMT, mitigating the risks associated with noisy data selection. While it significantly improves performance and training stability, challenges remain, such as automatically determining the optimal data cut-off point and addressing "catastrophic forgetting" of the original general-domain knowledge. Future research may explore multi-criteria curricula or apply these techniques to even more specialized and distant domains.

Summary updated

BibTeX
@inproceedings{zhang2019curriculum,
  author = {Xuan Zhang and Pamela Shapiro and Gaurav Kumar and Paul McNamee and Marine Carpuat and Kevin Duh},
  title = {{Curriculum Learning for Domain Adaptation in Neural Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=SyWfeX-uZB},
}
Figure