Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits

Julia Kreutzer, David Vilar, Artem Sokolov

Abstract

Training data for machine translation (MT) is often sourced from a multitude of large corpora that are multi-faceted in nature, e.g. containing contents from multiple domains or different levels of quality or complexity. Naturally, these facets do not occur with equal frequency, nor are they equally important for the test scenario at hand. In this work, we propose to optimize this balance jointly with MT model parameters to relieve system developers from manual schedule design. A multi-armed bandit is trained to dynamically choose between facets in a way that is most beneficial for the MT system. We evaluate it on three different multi-facet applications: balancing translationese and natural training data, or data from multiple domains or multiple language pairs. We find that bandit learning leads to competitive MT systems across tasks, and our analysis provides insights into its learned strategies and the underlying data sets.

Introduction

Modern Neural Machine Translation (NMT) models rely on massive, heterogeneous datasets. These "multi-faceted" datasets encompass various domains (e.g., legal, medical), multiple language pairs in multilingual settings, and varying data provenance (e.g., natural text vs. translationese).

The industry standard for managing these facets involves static heuristics, such as upsampling smaller datasets using a tuned temperature parameter. While common, these methods are expensive to tune, static throughout the training process, and often brittle when applied to new tasks. This paper proposes a dynamic alternative: using multi-armed bandits to automatically learn the optimal data mixture during training.

Methodology: The Bandit Framework

The authors frame data selection as a Multi-Armed Bandit (MAB) problem where each data facet is treated as an "arm." They employ the EXP3 algorithm, which is robust to adversarial environments where rewards may shift over time.

Bandit-driven data selection framework for NMT.

The bandit's objective is to maximize a reward signal based on the NMT model's learning progress. Several reward functions were evaluated, including:

  • Loss Drop: The reduction in training loss.
  • Prediction Gain (PG): The improvement in the model's ability to predict a specific batch after a gradient update.

A key finding was that calculating these rewards on a balanced development set—one that represents all facets equally—was significantly more effective than using training data. This approach directly aligns the bandit's strategy with the goal of achieving uniform performance across all facets.

Results and Analysis

The bandit-driven approach consistently outperformed static baselines. In the natural/translationese task, it achieved gains of 0.5 to 0.9 BLEU. In multi-domain settings, improvements reached up to +1.7 BLEU on specific domains, with an average increase of 1.1 points.

BLEU score progression across different bandit reward functions compared to baseline.

Efficiency and Convergence

As shown in the learning curves, bandit methods—particularly those using development set rewards (dev-pg, dev-pgnorm)—converge faster and reach higher final BLEU scores than the baseline. In multi-domain tasks, the bandit reached baseline performance levels using only 72% of the total iterations.

Learned Strategies: "Bandits Don't Follow Rules"

The bandit's learned schedules were often surprising. In multi-domain tasks, it frequently prioritized high-resource, diverse domains like Law and Subtitles, while largely ignoring low-resource ones like the Koran. Despite this, performance on the ignored domains did not suffer, suggesting the bandit successfully leveraged cross-domain transfer. In multilingual settings, the bandit independently "discovered" size-proportional sampling without explicit knowledge of dataset sizes.

Conclusion

This work demonstrates that multi-armed bandits can effectively automate the complex task of data balancing in NMT. By dynamically adjusting the data mix based on real-time learning progress, the MAB approach improves translation quality and training efficiency. Future work may explore contextual bandits to incorporate the NMT model's state more explicitly or adapt the framework for heterogeneous batch sampling.

Summary updated

Abstract

Training data for machine translation (MT) is often sourced from a multitude of large corpora that are multi-faceted in nature, e.g. containing contents from multiple domains or different levels of quality or complexity. Naturally, these facets do not occur with equal frequency, nor are they equally important for the test scenario at hand. In this work, we pro-pose to optimize this balance jointly with MT model parameters to relieve system developers from manual schedule design. A multi-armed bandit is trained to dynamically choose be-tween facets in a way that is most beneficial for the MT system. We evaluate it on three different multi-facet applications: balancing translationese and natural training data, or data from multiple domains or multiple language pairs.We find that bandit learning leads to competitive MT systems across tasks, and our analysis provides insights into its learned strategies and the underlying data sets
BibTeX
@misc{kreutzer2021bandits,
  author = {Julia Kreutzer and David Vilar and Artem Sokolov},
  title = {{Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits}},
  year = {2021},
  eprint = {2110.06997},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.06997},
}

Loading related content...

Figure