Self-Adversarial Learning with Comparative Discrimination for Text Generation

Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, Ming Zhou

Abstract

Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated samples. To address the issues, we propose a novel self-adversarial learning (SAL) paradigm for improving GANs' performance in text generation. In contrast to standard GANs that use a binary classifier as its discriminator to predict whether a sample is real or generated, SAL employs a comparative discriminator which is a pairwise classifier for comparing the text quality between a pair of samples. During training, SAL rewards the generator when its currently generated sentence is found to be better than its previously generated samples. This self-improvement reward mechanism allows the model to receive credits more easily and avoid collapsing towards the limited number of real samples, which not only helps alleviate the reward sparsity issue but also reduces the risk of mode collapse. Experiments on text generation benchmark datasets show that our proposed approach substantially improves both the quality and the diversity, and yields more stable performance compared to the previous GANs for text generation.

Introduction

Generative Adversarial Networks (GANs) have demonstrated remarkable success in continuous domains like image synthesis. However, applying GANs to discrete text generation remains challenging. Two primary obstacles hinder performance: reward sparsity and mode collapse. Reward sparsity occurs when a binary discriminator quickly learns to distinguish real from fake text, providing no useful gradient to the generator. Mode collapse occurs when the generator focuses on a narrow subset of the data distribution to "fool" the discriminator, sacrificing output diversity.

This paper introduces Self-Adversarial Learning (SAL), a paradigm shift that moves away from absolute "real vs. fake" classification toward relative quality comparison.

Methodology: Comparative Discrimination

The core of SAL is the Comparative Discriminator. Unlike standard discriminators that evaluate a single sample, the comparative discriminator takes a pair of sentences and classifies them into three categories: better ($>$), worse ($<$), or indistinguishable ($\approx$).

Comparison between Standard GANs (a) and the proposed Self-Adversarial Learning framework (b).

The SAL Framework

The SAL framework implements a self-play mechanism where the generator's current output is compared against its own historical outputs stored in a Memory Buffer.

  1. Self-Improvement Reward: The generator receives a positive reward if its current generation is judged superior to its past samples. This "moving goalpost" ensures that the generator always has a reachable target, mitigating reward sparsity.
  2. Scheduled Rewarding: Reward weights are adjusted over time to balance exploration and exploitation, becoming more conservative as the model matures.
  3. Memory Replay: A buffer of past generations prevents the model from overfitting to recent successes and ensures stable progress.

Experimental Results

The researchers evaluated SAL against several baselines, including SeqGAN, RankGAN, and MaliGAN, across synthetic data, Image COCO captions, and EMNLP2017 WMT News.

Quantitative Performance

SAL consistently outperformed all baselines in both quality and diversity. On the WMT News dataset, SAL achieved a Perplexity of 612.8, significantly lower than SeqGAN (743.8) and the Maximum Likelihood Estimation (MLE) baseline (702.6). Human evaluation further supported these findings, with SAL-generated text receiving a score of 3.84/5 on the COCO dataset.

Learning curves showing NLL_oracle (left) and Perplexity (right) for various models. SAL demonstrates superior convergence and lower final scores.

Stability and Diversity

As shown in the learning curves, SAL exhibits faster convergence and lower variance compared to standard GAN approaches. Furthermore, metrics for diversity (NLL_gen and FD) indicated that SAL effectively avoids mode collapse, maintaining a broader distribution of generated text than its competitors.

Conclusion

Self-Adversarial Learning addresses the fundamental limitations of text GANs by replacing binary rewards with a comparative, self-improvement mechanism. By competing against its own history rather than an absolute standard of "realness," the generator receives denser feedback and explores the data distribution more effectively. Future work may extend this comparative paradigm to other modalities, such as audio or image generation, where mode collapse remains a persistent challenge.

Summary updated

Abstract

Conventional Generative Adversarial Networks (GANs) for text generation tend to have issues of reward sparsity and mode collapse that affect the quality and diversity of generated samples. To address the issues, we propose a novel self-adversarial learning (SAL) paradigm for improving GANs' performance in text generation. In contrast to standard GANs that use a binary classifier as its discriminator to predict whether a sample is real or generated, SAL employs a comparative discriminator which is a pairwise classifier for comparing the text quality between a pair of samples. During training, SAL rewards the generator when its currently generated sentence is found to be better than its previously generated samples. This self-improvement reward mechanism allows the model to receive credits more easily and avoid collapsing towards the limited number of real samples, which not only helps alleviate the reward sparsity issue but also reduces the risk of mode collapse. Experiments on text generation benchmark datasets show that our proposed approach substantially improves both the quality and the diversity, and yields more stable performance compared to the previous GANs for text generation.
BibTeX
@inproceedings{zhou2020self,
  author = {Wangchunshu Zhou and Tao Ge and Ke Xu and Furu Wei and Ming Zhou},
  title = {{Self-Adversarial Learning with Comparative Discrimination for Text Generation}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=qKXeZbG6K7vE},
}
Figure