Text Generation by Learning from Off-Policy Demonstrations

Richard Yuanzhe Pang, He He

Abstract

Current approaches to text generation largely rely on autoregressive models and maximum likelihood estimation (MLE). This paradigm leads to (i) diverse but low-quality samples due to mismatched learning objectives and evaluation metrics (likelihood vs. quality) and (ii) exposure bias due to mismatched history distributions (gold vs. model-generated). To alleviate these problems, we frame text generation as an offline reinforcement learning (RL) problem with expert demonstrations, where the goal is to maximize quality given model-generated histories. We propose GOLD (generation by off-policy learning from demonstrations): an easy-to-optimize algorithm that learns from the demonstrations by importance weighting. Intuitively, GOLD upweights confident tokens and downweights unconfident ones in the reference during training, avoiding optimization issues faced by prior RL approaches that rely on online data collection. According to both automatic and human evaluation, models trained by GOLD outperform those trained by MLE and policy gradient on summarization, question generation, and machine translation. Further, our models are less sensitive to decoding algorithms and alleviate exposure bias.

Introduction

Most modern text generation systems are trained using Maximum Likelihood Estimation (MLE). While effective, MLE has two primary limitations. First, there is an objective mismatch: MLE maximizes the probability of the next word in a human reference, which does not always align with human-perceived quality metrics like fluency or correctness. This often results in "high recall but low precision," where the model learns to assign probability to many sequences, including low-quality ones. Second, exposure bias arises because models are trained on perfect "gold" histories but must rely on their own potentially flawed previous outputs during inference. This discrepancy can lead to compounding errors, often described as a "death spiral."

The GOLD Algorithm

To address these challenges, we propose GOLD (Generation by Off-policy Learning from Demonstrations). GOLD frames text generation as an offline reinforcement learning problem. Instead of requiring the model to explore an astronomically large search space—which often leads to instability in early training—GOLD learns strictly from a static set of human demonstrations.

The GOLD Algorithm uses importance weighting to stabilize offline RL training by upweighting confident tokens and downweighting unconfident ones in the reference.

Key components of GOLD include:

  • Off-Policy Policy Gradient: It uses importance weighting to estimate how an optimal policy would behave based on existing data.
  • Confidence Weighting: GOLD upweights tokens the model is already confident about in the reference and downweights those it finds unlikely. This encourages the model to focus on high-precision generations.
  • Novel Reward Functions: We experimented with several rewards, finding that a "sum of probabilities" ($R_s$) worked best by allowing for "partial credit," which helps the model recover from minor errors.

Evaluation and Analysis

We evaluated GOLD on four diverse benchmarks: Natural Question Generation (NQG), CNN/DailyMail news summarization, XSum extreme summarization, and IWSLT14 German-to-English translation.

Precision over Recall

A significant finding is that GOLD acts as a "perfectionist" compared to MLE's "completionist" approach. While MLE tries to learn every possible way a human might write a sentence, GOLD focuses on the most reliable, high-quality paths. This is reflected in the token-level negative log-likelihood (NLL) loss distribution; GOLD's loss is tightly concentrated near zero, whereas MLE's is more spread out across the dataset.

Histograms of token-level average NLL loss for NQG and CNN/DM tasks. GOLD variants (GOLD-p and GOLD-s) show a significantly higher concentration of low-loss tokens compared to the MLE baseline.

Performance and Robustness

GOLD consistently outperformed MLE and standard RL fine-tuning on automatic metrics like BLEU and ROUGE. For example, on NQG, GOLD achieved a BLEU score of 16.10 compared to MLE's 14.23. Human evaluators also preferred GOLD's outputs significantly more often. Furthermore, GOLD effectively mitigates exposure bias; while MLE quality often degrades as sentences get longer, GOLD remains stable. GOLD-trained models also perform well with simple greedy decoding, suggesting a cleaner internal probability distribution.

Conclusion

GOLD provides a robust framework for learning high-quality text generation from demonstrations. By prioritizing precision and utilizing importance weighting, it overcomes the inherent flaws of MLE and standard RL. While this focus on precision may slightly reduce output diversity, it is a valuable trade-off for tasks where accuracy and fluency are the primary goals.

Summary updated

Abstract

Current approaches to text generation largely rely on autoregressive models and maximum likelihood estimation. This paradigm leads to (i) diverse but low-quality samples due to mismatched learning objective and evaluation metric (likelihood vs. quality) and (ii) exposure bias due to mismatched history distributions (gold vs. model-generated). To alleviate these problems, we frame text generation as an offline reinforcement learning (RL) problem with expert demonstrations (i.e., the reference), where the goal is to maximize quality given model-generated histories. We propose GOLD (generation by off-policy learning from demonstrations): an easy-to-optimize algorithm that learns from the demonstrations by importance weighting. Intuitively, GOLD upweights confident tokens and downweights unconfident ones in the reference during training, avoiding optimization issues faced by prior RL approaches that rely on online data collection. According to both automatic and human evaluation, models trained by GOLD outperform those trained by MLE and policy gradient on summarization, question generation, and machine translation. Further, our models are less sensitive to decoding algorithms and alleviate exposure bias.
BibTeX
@inproceedings{pang2021text,
  author = {Richard Yuanzhe Pang and He He},
  title = {{Text Generation by Learning from Off-Policy Demonstrations}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=nTsnuGQEnD},
}
Figure