Improving Maximum Likelihood Training for Text Generation with Density Ratio Estimation

Yuxuan Song, Ning Miao, Hao Zhou, Lantao Yu, Mingxuan Wang, Lei Li

Abstract

Autoregressive neural sequence generative models trained by Maximum Likelihood Estimation (MLE) suffer from the exposure bias problem in practical finite sample scenarios. The crux is that the model is trained on ground-truth prefixes but must generate from its own potentially erroneous prefixes during inference. This paper introduces $\psi$-MLE, a framework that combines the stability of MLE with the exploratory power of Generative Adversarial Networks (GANs) using density ratio estimation.

Introduction

Most modern text generators are autoregressive, predicting the next token based on previous ones. Traditionally, these models are trained using MLE with "teacher forcing," where the model is always provided with the ground-truth prefix. However, during inference, the model relies on its own previous outputs. This discrepancy, known as exposure bias, causes small errors to accumulate, often leading to nonsensical or repetitive text.

While Sequential GANs (SeqGANs) attempt to address this by allowing the model to explore the entire sample space, they are notoriously difficult to train. They often suffer from high gradient variance and "mode collapse," where diversity is sacrificed for quality. $\psi$-MLE aims to bridge this gap through a self-augmentation strategy that maintains training stability while improving both quality and diversity.

The $\psi$-MLE Framework

$\psi$-MLE introduces a "self-augmentation" strategy to expand the model's supervision beyond the narrow training set. The framework consists of several key components:

  1. Mixture Distribution ($p_{mix}$): The model is trained on a mixture of real training data and samples generated by the model itself. This forces the generator to learn how to recover from its own mistakes, directly addressing exposure bias.
  2. Density Ratio Estimation: To ensure the model does not learn from low-quality "garbage" outputs, the authors introduce an importance weight ($r_\psi$). This weight is calculated using a density ratio estimator that determines the likelihood of a generated sample belonging to the target distribution.
  3. Bregman Divergence: Instead of a standard GAN classifier, $\psi$-MLE estimates the density ratio directly by minimizing the Bregman divergence. This approach provides a more stable and accurate training signal than the typical minimax game of GANs.
  4. Variance Reduction: Techniques such as self-normalization and ratio flattening are employed to keep the training process smooth and prevent the instabilities common in adversarial training.

Experimental Results

Synthetic Data and the Quality-Diversity Trade-off

The researchers first evaluated $\psi$-MLE on synthetic data generated by an "Oracle" LSTM. This setup allows for precise measurement of how well a model recovers a known underlying distribution.

Figure 1: Quality-diversity trade-off on synthetic data. $\psi$-MLE (red) achieves a better balance, maintaining lower NLL test (higher diversity) for the same NLL oracle (quality) compared to MLE and SeqGAN.

As shown in the figure above, $\psi$-MLE consistently achieves a superior trade-off between quality (NLL-oracle) and diversity (NLL-test) compared to standard MLE and SeqGAN baselines.

Real-World Performance

The framework was further tested on the EMNLP2017 News and Image COCO datasets. On the EMNLP News dataset, which contains complex, formal structures, $\psi$-MLE demonstrated significant improvements over existing state-of-the-art models.

Figure 3: Performance comparison on the EMNLP2017 News dataset. $\psi$-MLE consistently outperforms baselines in both quality (BLEU) and diversity (Self-BLEU).

On EMNLP News, $\psi$-MLE achieved a BLEU-5 score of 0.23 while maintaining a lower Self-BLEU score (0.36) than standard MLE, indicating that the generated text is both more accurate to the target domain and more diverse.

Conclusion

$\psi$-MLE successfully addresses the exposure bias problem by expanding the "zone of supervision" through self-augmentation and density ratio weighting. By moving away from the standard GAN minimax objective toward a density-ratio-weighted MLE objective, the authors achieve the exploratory benefits of GANs without sacrificing the stability of MLE. Future work will explore extending this framework to conditional text generation tasks such as machine translation and abstractive summarization.

Summary updated

Abstract

Autoregressive neural sequence generative models trained by Maximum Likelihood Estimation suffer the exposure bias problem in practical finite sample scenarios. The crux is that the number of train...
BibTeX
@inproceedings{song2020improving,
  author = {Yuxuan Song and Ning Miao and Hao Zhou and Lantao Yu and Mingxuan Wang and Lei Li},
  title = {{Improving Maximum Likelihood Training for Text Generation with Density Ratio Estimation}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=dkRoBAHayqV},
}
Figure