Abstract
Training data for machine translation (MT) is often sourced from a multitude of large corpora that are multi-faceted in nature, e.g. containing contents from multiple domains or different levels of quality or complexity. Naturally, these facets do not occur with equal frequency, nor are they equally important for the test scenario at hand. In this work, we propose to optimize this balance jointly with MT model parameters to relieve system developers from manual schedule design. A multi-armed bandit is trained to dynamically choose between facets in a way that is most beneficial for the MT system. We evaluate it on three different multi-facet applications: balancing translationese and natural training data, or data from multiple domains or multiple language pairs. We find that bandit learning leads to competitive MT systems across tasks, and our analysis provides insights into its learned strategies and the underlying data sets.
Introduction
Training a Neural Machine Translation (NMT) model effectively requires more than just a large volume of data; it requires the right data at the right time. Modern MT datasets are typically "multi-faceted," comprising a mix of domains (e.g., legal, medical), varying quality levels (human-translated vs. web-crawled), or multiple language pairs.
The standard approach to managing this diversity involves static heuristics, such as upsampling rare data or using temperature-based smoothing. However, these methods are brittle, requiring manual tuning and assuming a constant ideal data distribution throughout training. In practice, a model's data needs may evolve as it matures. This paper proposes using Multi-Armed Bandits (MAB) to dynamically select data facets, treating data selection as a reinforcement learning problem to discover optimal, non-intuitive training schedules.
Methodology
The authors frame data selection as a MAB problem where the system learns to select the most beneficial data "arms" based on feedback from the NMT model.
The Bandit Framework
- The Arms: Each arm represents a specific data facet (e.g., a domain or language pair).
- The Algorithm: The system employs EXP3, a bandit algorithm suited for adversarial settings where rewards may change unpredictably.
- The Reward Signal: Several signals based on model loss were evaluated:
- Plain Loss: Raw error value.
- Prediction Gain (pg): The decrease in loss after a training step.
- Relative Gain (pgnorm): The percentage improvement in loss.
Crucially, the most effective rewards were calculated on a small, balanced development set. This approach incentivizes the bandit to prioritize performance across all facets equally, rather than just minimizing training loss on the current batch.
Experimental Setup
The approach was evaluated across three distinct multi-facet scenarios:
- Natural vs. Translationese: Balancing original text against text previously translated, which often contains stylistic biases.
- Multi-domain: Balancing five domains (Medical, IT, Law, Koran, and Subtitles) for German-to-English translation.
- Multilingual: Handling many-to-one and one-to-many translations using the TED57 and OPUS100 (99 language pairs) datasets.
Results and Discussion
The bandit-led models consistently outperformed traditional "take-it-all" baselines and manual heuristics.
Key Findings
- Challenging Human Intuition: The learned schedules often contradicted human-designed curricula. While humans might start with "easy" data, bandits often preferred harder, high-diversity data early in the training process.
- Efficiency and Scalability: In multi-domain tasks, the bandit approach reached peak baseline performance in only 72% of the training time. On the massive OPUS100 dataset, it achieved a +1.2 BLEU improvement for English-to-X translations.
- Computational Efficiency: Unlike other dynamic selection methods that require expensive gradient calculations for every facet, the bandit approach is lightweight, requiring only a single extra forward pass on a small dev batch.
- Automatic Discovery: The bandit naturally learned to upsample low-resource languages in multilingual tasks, matching the performance of carefully tuned manual temperature sampling without prior knowledge of dataset sizes.
Conclusion
Multi-armed bandits offer a powerful, automated alternative to manual data scheduling in NMT. By dynamically adapting to the model's needs, they improve both translation quality and training efficiency. Future work may explore moving from "stateless" bandits to full Reinforcement Learning setups with state representations to unlock even more complex scheduling strategies, or refining the granularity of selection to individual sentences within batches.
Summary updated