Standard text generation models trained with Maximum Likelihood Estimation (MLE) often suffer from "exposure bias" and a mismatch between training objectives and human quality metrics. To address these issues, researchers introduce GOLD (Generation by Off-policy Learning from Demonstrations), an offline reinforcement learning algorithm that uses importance weighting to focus on high-precision generation. By learning directly from expert demonstrations without the instability of online exploration, GOLD produces models that are more robust, less sensitive to decoding strategies, and significantly better at maintaining quality in long sequences.
The Problem with Maximum Likelihood Estimation
Most modern text generators (like those behind translation tools or conversational agents) are autoregressive, predicting the next word based on previous words. Traditionally, these are trained using Maximum Likelihood Estimation (MLE). However, MLE has two major flaws:
- Objective Mismatch: MLE encourages "high recall"—the model tries to assign some probability to every possible valid word in the training data. Human evaluation, however, prizes "high precision"—we want the model to pick the best words. This mismatch often leads to models that over-generalize and produce low-quality, overly "safe" sequences.
- Exposure Bias: During training, the model is always shown the "gold" (correct) previous words. At inference time, it has to rely on its own previous (and potentially buggy) predictions. One small error can lead the model into a state it never saw during training, causing errors to compound into hallucinations or repetitive loops.
While Reinforcement Learning (RL) could theoretically bridge this gap by optimizing for sequence-level quality, standard online RL is notoriously difficult to train in NLP due to the exponentially large search space and sparse rewards.
Methods: Introducing GOLD
The authors propose GOLD (Generation by Off-policy Learning from Demonstrations). Instead of letting the model wander aimlessly through the space of all possible sentences (online exploration), GOLD treats text generation as an offline RL problem.
- Offline Learning: The model learns strictly from the "demonstrations" (the reference text in the dataset) without needing to interact with a live environment.
- Importance Weighting: GOLD uses an off-policy policy gradient. It upweights tokens in the reference that the current model is already confident about and downweights those it finds "confusing" or unlikely. This encourages the model to focus on "easy-to-learn" high-precision patterns.
- Reward Estimation: Rather than using a simple 0/1 reward for matching the reference, they use an estimated "perceptual quality" reward. They use a pre-trained MLE model to estimate how likely a human would be to produce a specific token given the context.
- Variants: The study tested three reward types: GOLD-$\delta$ (binary sequence-level reward), GOLD-$p$ (product of token probabilities), and GOLD-$s$ (sum of token probabilities). GOLD-$s$ generally performed best as it allows for "partial credit" even if the model makes a mistake at one step.
Datasets
The authors evaluated GOLD across four diverse text generation tasks:
- NQG (Natural Question Generation): Generating questions from a passage and an answer span (SQuAD dataset).
- CNN/DM: Standard news summarization.
- XSum (Extreme Summarization): Highly abstractive summarization of BBC news articles.
- IWSLT14 De-En: German-to-English machine translation.
Key Insights and Ideas
The Perplexity Paradox
Interestingly, GOLD models show higher held-out perplexity than MLE models. Usually, lower perplexity is seen as better. However, the authors show that GOLD’s high perplexity comes from a "long tail" of large losses on difficult or noisy examples it chooses to ignore.
By ignoring the noise and focusing on what it can do well, the model achieves higher precision and better human-rated quality despite the higher average negative log-likelihood.
Decoding Agnosticism and Exposure Bias
MLE models are very sensitive to how you search for words (e.g., beam search vs. greedy search). GOLD models are much more robust; they perform nearly as well with simple greedy decoding as they do with complex search algorithms because the policy itself is trained to consider future rewards.
Furthermore, the authors empirically demonstrate that as generation length increases, MLE quality plummets because the model "drifts" away from the training distribution. GOLD’s quality remains remarkably stable across long sequences, effectively alleviating the exposure bias problem.
Key Results
- Superior Metrics: GOLD outperformed both MLE and standard on-policy RL (Policy Gradient) across all tasks. For example, on NQG, GOLD-s achieved a 16.10 BLEU score compared to MLE's 14.23.
- Human Preference: In pairwise human evaluations, workers preferred GOLD-s outputs over MLE outputs significantly more often (e.g., 44.3% vs 23.0% on NQG).
- State-of-the-Art (SOTA): When applied to large pre-trained models like BART, GOLD achieved near-SOTA results on CNN/DM and XSum, showing that the method scales well to massive Transformer architectures.
Limitations and Future Work
- Diversity vs. Quality: GOLD prioritizes precision (quality) over recall (diversity). In creative writing tasks where you want many different valid outputs for one prompt, GOLD might suffer from "mode collapse," focusing only on the single most likely high-quality path.
- Reward Dependency: While GOLD uses a generic reward function ($p_{MLE}$), it still relies on a well-trained MLE model to provide those initial estimates.
- Future Directions: The authors suggest exploring ways to maintain GOLD's precision while re-introducing diversity, perhaps through more sophisticated entropy regularization or multi-objective rewards.
Summary updated