DORB: Dynamically Optimizing Multiple Rewards with Bandits

Ramakanth Pasunuru, Han Guo, Mohit Bansal

Abstract: Policy gradients-based reinforcement learning has proven to be a promising approach for directly optimizing non-differentiable evaluation metrics for language generation tasks. However, optimizing for a specific metric reward leads to improvements in mostly that metric only, suggesting that the model is gaming the formulation of that metric in a particular way without often achieving real qualitative improvements. Hence, it is more beneficial to make the model optimize multiple diverse metric rewards jointly. While appealing, this is challenging because one needs to manually decide the importance and scaling weights of these metric rewards. Further, it is important to consider using a dynamic combination and curriculum of metric rewards that flexibly changes over time. Considering the above aspects, in our work, we automate the optimization of multiple metric rewards simultaneously via a multi-armed bandit approach (DORB), where at each round, the bandit chooses which metric reward to optimize next, based on expected arm gains. We use the Exp3 algorithm for bandits and formulate two approaches for bandit rewards: (1) Single Multi-reward Bandit (SM-Bandit); (2) Hierarchical Multi-reward Bandit (HM-Bandit). We empirically show the effectiveness of our approaches via various automatic metrics and human evaluation on two important NLG tasks: question generation and data-to-text generation, including on an unseen-test transfer setup. Finally, we present interpretable analyses of the learned bandit curriculum over the optimized rewards.

Introduction

In Natural Language Generation (NLG), researchers often face a "metric gaming" problem: models optimized for a single metric like BLEU or ROUGE often find mathematical shortcuts to high scores without improving actual text quality. While multi-reward optimization can mitigate this, finding the optimal weights for multiple rewards is computationally expensive and typically assumes static importance throughout training. This paper introduces DORB (Dynamically Optimizing Multiple Rewards with Bandits), a framework that automates reward selection through a dynamic training curriculum.

Methodology

DORB treats the selection of which metric to optimize as a Multi-Armed Bandit (MAB) problem. Each "arm" represents a different reward function (e.g., BLEU, ROUGE, or Entailment). The system uses the Exp3 algorithm to adapt to changing reward distributions as the model learns.

Overview of the DORB framework showing the bandit-driven RL optimization loop.

The authors propose two specific architectures:

  • SM-Bandit (Single Multi-reward Bandit): A single bandit selects one metric per round, aiming to improve the average performance across all metrics.
  • HM-Bandit (Hierarchical Multi-reward Bandit): A two-level system where a top-level controller identifies underperforming metrics, and a second-level bandit selects the reward function most likely to address that specific weakness.

Experimental Results

The framework was evaluated on Question Generation (SQuAD) and Data-to-Text (WebNLG) tasks. In both cases, the bandit-based models outperformed standard Cross-Entropy and fixed-weight RL baselines.

Performance comparison on the Question Generation task (SQuAD dataset).

Key findings include:

  • Dynamic Curriculum: Analysis showed the bandit frequently switches between metrics, confirming that different training stages benefit from different optimization goals.
  • Generalization: The SM-Bandit performed particularly well on the "unseen" categories of the WebNLG dataset, suggesting that dynamic multi-reward optimization leads to more robust linguistic capabilities.
  • Human Alignment: Blind human evaluations rated DORB outputs as significantly more relevant and coherent than baseline models (p < 0.01).

Conclusion

DORB demonstrates that automating the training curriculum via multi-armed bandits is a highly effective strategy for multi-reward optimization in NLG. By moving away from hand-tuned heuristics, the system allows models to autonomously determine what they need to learn at any given moment, leading to better overall performance and human-rated quality.

Summary updated

Abstract

Policy gradients-based reinforcement learning has proven to be a promising approach for directly optimizing non-differentiable evaluation metrics for language generation tasks. However, optimizing for a specific metric reward leads to improvements in mostly that metric only, suggesting that the model is gaming the formulation of that metric in a particular way without often achieving real qualitative improvements. Hence, it is more beneficial to make the model optimize multiple diverse metric rewards jointly. While appealing, this is challenging because one needs to manually decide the importance and scaling weights of these metric rewards. Further, it is important to consider using a dynamic combination and curriculum of metric rewards that flexibly changes over time. Considering the above aspects, in our work, we automate the optimization of multiple metric rewards simultaneously via a multi-armed bandit approach (DORB), where at each round, the bandit chooses which metric reward to optimize next, based on expected arm gains. We use the Exp3 algorithm for bandits and formulate two approaches for bandit rewards: (1) Single Multi-reward Bandit (SM-Bandit); (2) Hierarchical Multi-reward Bandit (HM-Bandit). We empirically show the effectiveness of our approaches via various automatic metrics and human evaluation on two important NLG tasks: question generation and data-to-text generation, including on an unseen-test transfer setup. Finally, we present interpretable analyses of the learned bandit curriculum over the optimized rewards.
BibTeX
@inproceedings{pasunuru2020dorb,
  author = {Ramakanth Pasunuru and Han Guo and Mohit Bansal},
  title = {{DORB: Dynamically Optimizing Multiple Rewards with Bandits}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=OwErfuwJgx1},
}
Figure