Adversarial Generation of Natural Language

Sai Rajeswar, Sandeep Subramanian, Francis Dutil, Christopher Joseph Pal, Aaron C. Courville

Introduction

Most natural language generation (NLG) models are traditionally trained using Maximum Likelihood Estimation (MLE) with "teacher forcing." While effective for many tasks, this approach suffers from exposure bias: because the model only sees ground-truth context during training, it never learns how to recover from its own errors during inference.

Generative Adversarial Networks (GANs) offer a potential solution by training on entire sequences. However, they face a fundamental hurdle: text is discrete. In a standard GAN, the generator samples a word, but the sampling operation is non-differentiable, preventing gradients from the discriminator from flowing back to the generator. While previous works attempted to solve this using Reinforcement Learning (RL) estimators like Policy Gradients, these methods are notoriously unstable and high-variance.

Bypassing the Sampling Bottleneck

This paper introduces a surprisingly simple yet effective baseline to bypass the non-differentiability of discrete sampling. Instead of forcing the generator to sample a specific token, the authors propose a continuous relaxation where the discriminator operates directly on probability distributions.

The Continuous Softmax Approach

As illustrated in the architecture below, the generator passes its entire softmax probability distribution—a vector of probabilities over the whole vocabulary—for each time step to the discriminator. The discriminator then compares these continuous distributions against "one-hot" vectors (where the probability is 1 for the actual word and 0 elsewhere) representing the real data.

Architecture of the proposed text GAN using continuous softmax distributions.

To make this work, the authors leverage Wasserstein GANs with Gradient Penalty (WGAN-GP). Standard GANs often fail in this setup because the discriminator can easily "cheat" by identifying the sparsity of one-hot vectors versus the relative density of a softmax distribution. WGAN-GP provides a smoother metric and more stable gradients, preventing the discriminator from saturating too early and ensuring the generator receives useful feedback.

Architectural Refinements

The researchers experimented with both LSTMs and CNNs. For LSTM generators, they found it essential to include "peephole" connections—feeding the previous output distribution directly into the next hidden state. This helps the model maintain auto-regressive coherence, compensating for the fact that it isn't "sampling" a single word to condition on during the training phase.

Key Results and Evaluation

The models were evaluated across synthetic grammars, natural language datasets (like the Penn Treebank), and specialized tasks like Chinese poetry generation.

Chinese Poetry Generation

The model achieved state-of-the-art results in Chinese poetry generation, specifically for 4-line poems with 5 or 7 characters per line. As shown in the table below, the LSTM-based GAN models outperformed established RL-based approaches like SeqGAN and MaliGAN in terms of BLEU scores, demonstrating that the continuous relaxation is not just simpler, but more effective.

Comparison of BLEU scores for Chinese poetry generation across different models.

Grammar Adherence and Conditional Synthesis

On synthetic Context-Free Grammar (CFG) tasks, the WGAN-GP models achieved nearly 99% accuracy in generating valid strings for short sequences. Furthermore, the authors successfully demonstrated conditional text synthesis. By concatenating attribute maps to the latent noise, the model could specifically generate questions versus statements or flip the sentiment of a generated sentence.

Discussion and Future Work

The most significant insight of this work is that complex RL-based sampling is not a prerequisite for training text GANs. Operating in the continuous softmax space provides a direct path for gradients, significantly simplifying the training pipeline.

However, limitations remain. Like many GAN architectures, these models still struggle with "mode dropping," where the generator captures only a subset of the diversity found in the real data. Additionally, while CNNs handled longer sequences well, LSTMs required careful curriculum learning—starting with short sequences and gradually increasing length—to avoid training collapse. Future research may explore these GANs for non-goal-oriented dialog systems, where traditional MLE often produces dull, repetitive responses.

Summary updated

BibTeX
@inproceedings{rajeswar2017adversarial,
  author = {Sai Rajeswar and Sandeep Subramanian and Francis Dutil and Christopher Joseph Pal and Aaron C. Courville},
  title = {{Adversarial Generation of Natural Language}},
  booktitle = {Rep4NLP @ ACL},
  year = {2017},
  doi = {10.18653/v1/w17-2629},
}
Figure