Introduction
Neural Machine Translation (NMT) models have achieved remarkable success but remain notoriously "brittle." While they perform exceptionally well on data similar to their training set, performance often collapses when faced with a new domain—for instance, a model trained on news text struggling with legal patents.
The standard approach to this issue is Domain Adaptation, typically through "continued training" or fine-tuning. In this process, a general-purpose model is further trained on a small set of in-domain data. To enhance this, researchers often supplement the training with "pseudo" in-domain sentences selected from massive, noisy web crawls like Paracrawl. However, a significant challenge remains: how much extra data should be included? Too little data misses helpful vocabulary, while too much introduces noise that degrades performance.
Probabilistic Curriculum Learning
To address the data selection trade-off, Zhang et al. propose a Probabilistic Curriculum Learning (CL) strategy. Instead of treating all selected data as equally relevant, they structure the training process like a school syllabus, introducing data in a "most-similar-to-least-similar" schedule.
The method involves four key steps:
- Similarity Scoring: Sentences from a general web crawl are ranked based on their similarity to the target domain using methods like Moore-Lewis (cross-entropy differences) and Cynical Data Selection.
- Sharding: The ranked sentences are grouped into "shards."
- Phased Training: Training is divided into phases. Initially, the model only sees the most similar shard. In subsequent phases, less similar shards are gradually added to the training pool.
- Probabilistic Sampling: To maintain the benefits of randomization essential for deep learning, the approach shuffles data within the current pool and employs standard practices like bucketing by sentence length.
Experimental Setup
The researchers evaluated their approach across four adaptation tasks involving two language pairs: German-English (de-en) and Russian-English (ru-en). The target domains were TED Talk transcripts and Patents from the World International Property Organization. General domain data comprised approximately 28 million pairs for German and 51 million for Russian, while Paracrawl provided the source for unlabeled web data.
Key Results and Insights
The curriculum approach consistently outperformed standard continued training across all domains and language pairs, achieving gains of up to 3.22 BLEU points over the adapted baseline.
Robustness to Noise
A critical finding was the model's robustness to noise. In standard continued training, performance often peaks and then declines as more web-crawled data is added because the "tail" of the data is of lower quality. In contrast, the CL models maintained high performance even when adding up to 4 million extra sentences, effectively buffering against the negative impact of less relevant data.
Lexical Choice and Error Analysis
Using $S^4$ error analysis, the authors discovered that the curriculum does more than just help the model learn new vocabulary (SEEN errors). Its primary benefit lies in helping the model choose the correct meaning for words it already knows but which have different nuances in the new domain (SCORE and SENSE errors). This suggests that the structured introduction of data helps the model refine its internal representations for domain-specific contexts.
Conclusion
Curriculum learning provides a robust framework for domain adaptation in NMT, mitigating the risks associated with noisy data selection. While it significantly improves performance and training stability, challenges remain, such as automatically determining the optimal data cut-off point and addressing "catastrophic forgetting" of the original general-domain knowledge. Future research may explore multi-criteria curricula or apply these techniques to even more specialized and distant domains.
Summary updated