TaylorGAN: Neighbor-Augmented Policy Update Towards Sample-Efficient Natural Language Generation

Chun-Hsing Lin, Siang-Ruei Wu, Hung-yi Lee, Yun-Nung ChenNeurIPS 2020 · OpenReview

TL;DR

TaylorGAN introduces a novel approach to training Generative Adversarial Networks (GANs) for natural language generation. By leveraging a first-order Taylor expansion, the model "peeks" at the discriminator's gradients. This allows it to learn from neighboring sentences without extra computational overhead, enabling stable training from scratch with high sample efficiency and state-of-the-art results.

The Problem: The "Discrete Bottleneck"

Training GANs for Natural Language Generation (NLG) is notoriously difficult because of the discrete nature of text. In image generation, pixels are continuous; you can slightly nudge a pixel's value and observe how the discriminator reacts. In text, tokens (words) are discrete. You cannot "slightly" change the word "cat" to "car" in a differentiable way—you either sample one or the other.

To bypass this, researchers typically treat the discriminator as a "black box" reward provider and use Reinforcement Learning (RL) techniques like REINFORCE. However, REINFORCE is "blind." It only knows if an entire generated sentence was good or bad, but not why. This lack of granular feedback leads to high variance, poor sample efficiency, and the need for massive batch sizes or "warm-starting" the model with Maximum Likelihood Estimation (MLE) pre-training.

Methods: The TaylorGAN Approach

The authors propose TaylorGAN, which treats the discriminator not as a black box, but as a differentiable guide. Its core components include:

1. The Taylor Estimator

Instead of just calculating the reward for a single generated sentence, the authors use a first-order Taylor expansion in the embedding space. This approximates the rewards for "neighboring" sentences using the gradient information already available from the discriminator.

Comparison of REINFORCE and Taylor estimators.

2. Hamming Transition

They define a neighborhood based on the "unit Hamming sphere"—essentially all sentences that can be reached by changing exactly one word. By looking at these neighbors, the generator receives a much denser signal about which specific word substitutions would improve the overall score.

Evaluating neighboring sentences using Taylor expansion.

3. Discriminator Constraints

For a Taylor expansion to be accurate, the reward function must be relatively smooth. The authors enforce this by using Spectral Normalization and embedding constraints. This ensures the discriminator is Lipschitz continuous, meaning its output doesn't jump wildly for small changes in the input.

4. Off-Policy Updates

Because the model evaluates neighbors that weren't actually sampled by the current policy, it uses importance sampling to correctly update the generator's policy (an off-policy update).

Key Insights

  • The Discriminator is a Map, Not Just a Judge: Standard RL-based GANs ignore the internal "knowledge" of the discriminator. TaylorGAN shows that the gradient of the discriminator with respect to the input embeddings points toward "better" words, providing crucial directional guidance for optimization.
  • Interpolating Between Estimators: By tuning a "bandwidth" parameter, TaylorGAN can interpolate between the high-variance/unbiased REINFORCE estimator and the low-variance/biased Straight-Through estimator, finding a "sweet spot" for training stability.
  • Efficiency Over Brute Force: Unlike previous "from scratch" models (like ScratchGAN) that require massive batch sizes (e.g., 512 or 1024) to reduce variance, TaylorGAN achieves superior results with a modest batch size of just 64.

Key Results

Evaluated primarily on the EMNLP 2017 News dataset (and COCO image captions in the appendix), TaylorGAN demonstrated impressive capabilities:

  • State-of-the-Art Performance: TaylorGAN outperforms existing GAN-based methods (like SeqGAN, MaliGAN, and RelGAN) across multiple metrics, including BLEU (quality) and Self-BLEU (diversity).
  • Superior Global Consistency: On the Fréchet Embedding Distance (FED) metric—which measures how similar the distribution of generated text is to real text—TaylorGAN achieved a score of 0.0149, significantly better than MLE (0.0194) and other GAN variants.
  • No Pre-training Required: Most text GANs fail if they aren't pre-trained with MLE. TaylorGAN can be trained from a random initialization and still reach high performance, proving its robustness and sample efficiency.

Limitations and Future Work

The authors acknowledge a specific bias in their current implementation: when estimating the reward for a "neighboring" word at step $t$, they ignore how that change might affect the auto-regressive generation of all subsequent words ($t+1$ to $T$). While they provide a theoretical way to eliminate this bias, they leave the practical implementation for future work due to added complexity.

Additionally, while TaylorGAN is highly efficient, it still requires careful tuning of hyperparameters like the Lipschitz constraint and entropy maximization to prevent mode collapse. Future research could explore applying this Taylor-based gradient augmentation to broader model-based reinforcement learning tasks.

Summary updated

Abstract

Score function-based natural language generation (NLG) approaches such as REINFORCE, in general, suffer from low sample efficiency and training instability problems. This is mainly due to the non-differentiable nature of the discrete space sampling and thus these methods have to treat the discriminator as a black box and ignore the gradient information. To improve the sample efficiency and reduce the variance of REINFORCE, we propose a novel approach, TaylorGAN, which augments the gradient estimation by off-policy update and the first-order Taylor expansion. This approach enables us to train NLG models from scratch with smaller batch size --- without maximum likelihood pre-training, and outperforms existing GAN-based methods on multiple metrics of quality and diversity.
BibTeX
@inproceedings{lin2020taylorgan,
  author = {Chun-Hsing Lin and Siang-Ruei Wu and Hung-yi Lee and Yun-Nung Chen},
  title = {{TaylorGAN: Neighbor-Augmented Policy Update Towards Sample-Efficient Natural Language Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=FynD8vuyZ4u},
}
Figure