Abstract
Current approaches to text generation largely rely on autoregressive models and maximum likelihood estimation (MLE). This paradigm leads to (i) diverse but low-quality samples due to mismatched learning objectives and evaluation metrics (likelihood vs. quality) and (ii) exposure bias due to mismatched history distributions (gold vs. model-generated). To alleviate these problems, we frame text generation as an offline reinforcement learning (RL) problem with expert demonstrations, where the goal is to maximize quality given model-generated histories. We propose GOLD (generation by off-policy learning from demonstrations): an easy-to-optimize algorithm that learns from the demonstrations by importance weighting. Intuitively, GOLD upweights confident tokens and downweights unconfident ones in the reference during training, avoiding optimization issues faced by prior RL approaches that rely on online data collection. According to both automatic and human evaluation, models trained by GOLD outperform those trained by MLE and policy gradient on summarization, question generation, and machine translation. Further, our models are less sensitive to decoding algorithms and alleviate exposure bias.
Introduction
Most modern text generation systems are trained using Maximum Likelihood Estimation (MLE). While effective, MLE has two primary limitations. First, there is an objective mismatch: MLE maximizes the probability of the next word in a human reference, which does not always align with human-perceived quality metrics like fluency or correctness. This often results in "high recall but low precision," where the model learns to assign probability to many sequences, including low-quality ones. Second, exposure bias arises because models are trained on perfect "gold" histories but must rely on their own potentially flawed previous outputs during inference. This discrepancy can lead to compounding errors, often described as a "death spiral."
The GOLD Algorithm
To address these challenges, we propose GOLD (Generation by Off-policy Learning from Demonstrations). GOLD frames text generation as an offline reinforcement learning problem. Instead of requiring the model to explore an astronomically large search space—which often leads to instability in early training—GOLD learns strictly from a static set of human demonstrations.
Key components of GOLD include:
- Off-Policy Policy Gradient: It uses importance weighting to estimate how an optimal policy would behave based on existing data.
- Confidence Weighting: GOLD upweights tokens the model is already confident about in the reference and downweights those it finds unlikely. This encourages the model to focus on high-precision generations.
- Novel Reward Functions: We experimented with several rewards, finding that a "sum of probabilities" ($R_s$) worked best by allowing for "partial credit," which helps the model recover from minor errors.
Evaluation and Analysis
We evaluated GOLD on four diverse benchmarks: Natural Question Generation (NQG), CNN/DailyMail news summarization, XSum extreme summarization, and IWSLT14 German-to-English translation.
Precision over Recall
A significant finding is that GOLD acts as a "perfectionist" compared to MLE's "completionist" approach. While MLE tries to learn every possible way a human might write a sentence, GOLD focuses on the most reliable, high-quality paths. This is reflected in the token-level negative log-likelihood (NLL) loss distribution; GOLD's loss is tightly concentrated near zero, whereas MLE's is more spread out across the dataset.
Performance and Robustness
GOLD consistently outperformed MLE and standard RL fine-tuning on automatic metrics like BLEU and ROUGE. For example, on NQG, GOLD achieved a BLEU score of 16.10 compared to MLE's 14.23. Human evaluators also preferred GOLD's outputs significantly more often. Furthermore, GOLD effectively mitigates exposure bias; while MLE quality often degrades as sentences get longer, GOLD remains stable. GOLD-trained models also perform well with simple greedy decoding, suggesting a cleaner internal probability distribution.
Conclusion
GOLD provides a robust framework for learning high-quality text generation from demonstrations. By prioritizing precision and utilizing importance weighting, it overcomes the inherent flaws of MLE and standard RL. While this focus on precision may slightly reduce output diversity, it is a valuable trade-off for tasks where accuracy and fluency are the primary goals.
Summary updated