DORB: Dynamically Optimizing Multiple Rewards with Bandits

Ramakanth Pasunuru, Han Guo, Mohit Bansal

Abstract

Training Natural Language Generation (NLG) models often involves optimizing for a single metric, which can lead to "metric gaming" where models exploit statistical shortcuts at the expense of overall text quality. We present DORB (Dynamically Optimizing Multiple Rewards with Bandits), a framework that uses a multi-armed bandit to automatically select and balance multiple evaluation metrics during training. By creating a dynamic curriculum, DORB improves performance across tasks such as question generation and data-to-text conversion, achieving higher accuracy and more natural language as validated by both automated metrics and human evaluation.

Introduction

In Natural Language Generation (NLG) tasks—such as summarization or question generation—models are traditionally trained using cross-entropy loss. However, this approach often suffers from exposure bias, where the model fails to generalize because it is never exposed to its own errors during training.

To mitigate this, researchers frequently employ Reinforcement Learning (RL) to optimize models directly for evaluation metrics like BLEU or ROUGE. However, optimizing for a single metric often results in "metric gaming," where the model finds statistical shortcuts to boost scores without improving actual text quality. While multi-reward optimization is possible, manually balancing weights is a difficult, static process. DORB addresses this by automating reward selection through a dynamic bandit-based approach.

Methodology

The DORB framework treats the selection of which metric to optimize as a Multi-Armed Bandit (MAB) problem. This allows the model to adaptively focus on different aspects of quality throughout the training process.

Overview of the DORB framework showing the interaction between the Multi-Armed Bandit and the RL optimization process.

The Bandit Mechanism

DORB utilizes the Exp3 algorithm, designed for adversarial settings where rewards are non-stationary. At each training round, the bandit "pulls an arm" to select a specific metric. To ensure compatibility between metrics with different scales (e.g., 0–1 vs. 0–100), the framework employs a quantile-based scaling method to normalize all rewards into a [0, 1] range.

Architectures

The authors propose two primary variants:

  • SM-Bandit (Single Multi-reward): A single bandit optimizes based on the average of all scaled metrics.
  • HM-Bandit (Hierarchical Multi-reward): A hierarchical structure where a top-level controller identifies underperforming metrics, and a child bandit determines the best optimization path for that specific metric.

Experimental Setup

The researchers evaluated DORB on two distinct NLG tasks:

  1. Question Generation (QG): Using the SQuAD dataset, optimizing for ROUGE-L, Paraphrase Probability (QPP), and Answerability (QAP).
  2. Data-to-Text (D2T): Using the WebNLG dataset, optimizing for BLEU, ROUGE-L, and an Entailment-Score to ensure factual correctness.

Results and Discussion

DORB consistently outperformed standard cross-entropy baselines and static multi-reward RL models across both tasks.

Performance comparison of DORB variants against baselines on the Question Generation (SQuAD) task.

Quantitative Performance

On the SQuAD dataset, the SM-Bandit achieved a BLEU-4 score of 18.68, significantly higher than the cross-entropy baseline (17.88). In the WebNLG task, the models improved across all metrics, notably reducing the Translation Edit Rate (TER) to 33.58 compared to the baseline's 33.97.

Human Evaluation and Insights

Human judges rated DORB's outputs as significantly more relevant and coherent than single-reward models. This confirms that the bandit approach successfully avoids the metric gaming trap by learning an automated "curriculum." For instance, the bandit might prioritize stabilizing basic word overlap (ROUGE) before attempting complex semantic rewards like answerability.

Conclusion

DORB demonstrates that dynamic reward selection via multi-armed bandits leads to superior NLG performance and more human-like text. While the approach introduces some computational overhead due to frequent validation, it eliminates the need for manual weight tuning. Future work may explore contextual bandits to tailor reward selection to specific input properties, further refining the generation process.

Summary updated

BibTeX
@inproceedings{pasunuru2020dorb,
  author = {Ramakanth Pasunuru and Han Guo and Mohit Bansal},
  title = {{DORB: Dynamically Optimizing Multiple Rewards with Bandits}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=OwErfuwJgx1},
}
Figure