Improving Maximum Likelihood Training for Text Generation with Density Ratio Estimation

Yuxuan Song, Ning Miao, Hao Zhou, Lantao Yu, Mingxuan Wang, Lei Li

TL;DR

This paper introduces $\psi$-MLE, a novel training scheme for autoregressive text generation models that tackles the "exposure bias" problem inherent in Maximum Likelihood Estimation (MLE). By combining self-augmentation with density ratio estimation for bias correction, $\psi$-MLE achieves superior quality and diversity in generated text compared to existing state-of-the-art methods.

What Problem Does This Paper Address?

Autoregressive models, which generate sequences token by token, are commonly trained using Maximum Likelihood Estimation (MLE). However, MLE faces two significant challenges in practical scenarios, especially for text generation:

  1. Exposure Bias: During training, these models are typically trained using "teacher forcing," where they always receive the correct previous token as input. At inference time, however, they must use their own generated tokens as input. If the model makes a mistake early on, this error can cascade, leading to increasingly nonsensical outputs. This discrepancy between training and inference is known as exposure bias.
  2. Limited Training Data & Overfitting: In finite sample scenarios, the training dataset is limited. MLE models can overfit to this limited data, struggling to generalize to the vast space of possible text sequences.

Existing non-MLE methods, like GANs, attempt to address these issues but often suffer from high variance during training or can lead to mode collapse, sacrificing diversity for quality.

Methods

The paper proposes $\psi$-MLE, a new training scheme that aims to combine the strengths of MLE (direct training on high-quality samples) and GANs (exploration of unseen data). The core ideas are:

  • Self-Augmentation & Mixed Distribution: $\psi$-MLE augments the training data by creating a mixture distribution $P_{mix}(S) = m P_{data}(S) + (1-m) P_{\theta}(S)$, where $P_{data}$ is the true data distribution, $P_{\theta}$ is the model's current distribution, and $m$ is a mixing weight. This effectively expands the "effect zone of supervision" to include samples generated by the model itself, helping to alleviate exposure bias and overfitting.
  • Density Ratio Estimation for Bias Correction: While mixing distributions helps, directly training on $P_{mix}$ can still be problematic if $P_{\theta}$ is far from $P_{data}$ early in training, leading to noisy samples. To correct this, $\psi$-MLE uses density ratio estimation to learn a weight $r_{\psi}(S)$ that represents the ratio $P_{data}(S) / P_{mix}(S)$. This weight is then used in an importance sampling scheme to re-weight the samples from $P_{mix}$, effectively pushing the training towards the true data distribution.
  • Bregman Divergence for Density Ratio: The density ratio estimator $r_{\psi}$ is trained by minimizing a Bregman divergence between the estimated and true density ratios. This is framed as a classification problem where samples from $P_{data}$ are positive and samples from $P_{mix}$ are negative, and a classifier learns to distinguish them. The output of this classifier is then used to derive the density ratio.

The overall training procedure involves alternating updates between the generator ($\theta$) and the density ratio estimator ($\psi$), as outlined in Algorithm 1.

Datasets

The paper evaluates $\psi$-MLE on:

  • Synthetic Data: Used to analyze the model's behavior under controlled conditions and compare it with baselines.
  • Image COCO Captions: A dataset of short image descriptions.
  • EMNLP 2017 News: A dataset of longer formal texts.

No new benchmarks were introduced.

Key Insights and Ideas

  • Bridging MLE and GANs: The paper successfully demonstrates a principled way to combine the stability of MLE with the exploration capabilities of GANs, creating a more robust training framework.
  • Density Ratio as a Correction Mechanism: Using density ratio estimation to correct for the bias introduced by sampling from the model's own distribution is a key innovation. It allows the model to learn from its own generated data without being overly penalized by potentially low-quality samples.
  • The Importance of the "Effect Zone of Supervision": The concept of the "effect zone" helps to explain why different models perform differently. $\psi$-MLE expands this zone by incorporating self-generated samples, leading to better generalization.
  • Mixture Weight is Crucial: The experiments show that an intermediate mixture weight ($m$) is optimal, suggesting a balance between leveraging the true data distribution and exploring the model's generated distribution.

Key Results

  • Improved Quality and Diversity: $\psi$-MLE consistently outperforms MLE and state-of-the-art GAN-based methods (SeqGAN, LeakGAN, MaliGAN, COT) on both synthetic and real-world text generation tasks in terms of quality (measured by BLEU scores and NLL) and diversity (measured by Self-BLEU).
  • Stable Training: The proposed method is shown to be more stable during training compared to some GAN-based approaches, particularly when using specific objective functions for density ratio estimation.
  • Synthetic Turing Test: On a synthetic Turing test, $\psi$-MLE achieved the best likelihood-based benchmark results, indicating its ability to generate data that is indistinguishable from real data.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper explores the mixture weight $m$, other hyperparameters related to the density ratio estimation and the objective functions might require careful tuning.
  • Computational Cost: Density ratio estimation can add computational overhead to the training process.
  • Future Work: The authors suggest extending the $\psi$-MLE paradigm to conditional text generation tasks like machine translation, dialogue systems, and abstractive summarization. They also aim to further investigate the theoretical properties of their approach, such as consistency and generalization.

Summary updated

Abstract

Auto-regressive sequence generative models trained by Maximum Likelihood Estimation suffer the exposure bias problem in practical finite sample scenarios. The crux is that the number of training samples for Maximum Likelihood Estimation is usually limited and the input data distributions are different at training and inference stages. Many method shave been proposed to solve the above problem (Yu et al., 2017; Lu et al., 2018), which relies on sampling from the non-stationary model distribution and suffers from high variance or biased estimations. In this paper, we propose{\psi}-MLE, a new training scheme for auto-regressive sequence generative models, which is effective and stable when operating at large sample space encountered in text generation. We derive our algorithm from a new perspective of self-augmentation and introduce bias correction with density ratio estimation. Extensive experimental results on synthetic data and real-world text generation tasks demonstrate that our method stably outperforms Maximum Likelihood Estimation and other state-of-the-art sequence generative models in terms of both quality and diversity.
BibTeX
@inproceedings{song2020improving,
  author = {Yuxuan Song and Ning Miao and Hao Zhou and Lantao Yu and Mingxuan Wang and Lei Li},
  title = {{Improving Maximum Likelihood Training for Text Generation with Density Ratio Estimation}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=dkRoBAHayqV},
}
Figure