Abstract
To train neural machine translation models simultaneously on multiple tasks (languages), it is common to sample each task uniformly or in proportion to dataset sizes. As these methods offer little control over performance trade-offs, we explore different task scheduling approaches. We first consider existing non-adaptive techniques, then move on to adaptive schedules that over-sample tasks with poorer results compared to their respective baseline. As explicit schedules can be inefficient, especially if one task is highly over-sampled, we also consider implicit schedules, learning to scale learning rates or gradients of individual tasks instead. These techniques allow training multilingual models that perform better for low-resource language pairs (tasks with small amount of data), while minimizing negative effects on high-resource tasks.
Introduction
In Neural Machine Translation (NMT), the goal is often to develop a single "universal" model capable of handling multiple language pairs. However, training such multi-task models frequently results in a performance "tug-of-war." High-resource tasks, which have abundant data, compete for model capacity with low-resource tasks. Without careful management, the model may suffer from catastrophic forgetting of low-resource languages or hit capacity bottlenecks that hinder performance on high-resource ones.
The Limitations of Static Sampling
Standard training regimes typically rely on fixed sampling schedules. Uniform sampling treats all tasks equally, which often leads to overfitting on small datasets. Conversely, proportional sampling favors tasks based on their dataset size, which can cause the model to ignore low-resource languages entirely.
As shown in the experimental results for fixed ratios, there is a clear trade-off: increasing the sampling frequency of a high-resource task like English-French (En-Fr) improves its performance but eventually degrades the performance of the lower-resource English-German (En-De) task once a certain threshold is passed.
Adaptive Scheduling Framework
To address these limitations, we propose adaptive scheduling, which dynamically adjusts task priority based on real-time validation performance (BLEU scores). We explore two primary implementations:
- Explicit Schedules: These modify the probability of a task being sampled for the next training batch. If a task's performance lags behind its single-task baseline, its sampling probability is increased.
- Implicit Schedules: Rather than changing the sampling frequency, this method adjusts the "importance" of each task by scaling its learning rates or gradients.
A critical technical challenge for implicit scheduling is the interaction with the Adam optimizer. Because Adam normalizes updates by the square root of the second moment, it is largely scale-insensitive. To ensure that adaptive weights effectively influence the model, we utilize distinct optimizers for different tasks with independently scaled learning rates.
Results and Robustness
Our experiments on the WMT’14 benchmark (En-Fr as high-resource, En-De as low-resource) demonstrate that adaptive methods can automatically find a near-optimal balance. Explicit adaptive scheduling achieved a BLEU score of 26.35 on En-De (a +1.45 gain over the single-task baseline) while maintaining En-Fr performance at 40.70, nearly matching its standalone baseline of 40.80.
Furthermore, these adaptive methods show significant robustness to hyperparameter choices. As illustrated in the heatmaps below, performance remains relatively stable across various settings for $\alpha$ (which controls the aggressiveness of the adaptation) and $\epsilon$ (which ensures numerical stability).
Conclusion
Adaptive scheduling provides a powerful, automated alternative to the manual tuning of task sampling ratios in multi-task NMT. By using validation performance as a feedback signal, these methods protect low-resource tasks from being forgotten while minimizing the negative impact on high-resource tasks. Future work will focus on scaling these techniques to massively multilingual models involving dozens or hundreds of languages, where manual tuning is no longer feasible.
Summary updated