InitialGAN: A Language GAN With Completely Random Initialization

Da Ren, Qing Li

TL;DR

This paper introduces InitialGAN, a novel language GAN that achieves state-of-the-art performance without relying on pre-training techniques, a common bottleneck for previous language GANs. By addressing issues of "invalid sampling" and "unhealthy gradients" with new techniques like dropout sampling and a fully normalized LSTM, InitialGAN demonstrates that language GANs can be trained from scratch and outperform traditional MLE methods.

What Problem Does This Paper Address?

Traditional text generation models often suffer from the "exposure bias" problem, where discrepancies between training (using ground truth) and inference (using generated tokens) lead to degraded sentence quality. Generative Adversarial Networks (GANs) offer a promising solution by providing a consistent generation process. However, applying GANs to text generation is challenging due to the non-differentiable nature of sampling word probabilities. Existing GAN approaches either use high-variance estimators like REINFORCE or biased continuous relaxations, both of which necessitate pre-training (e.g., MLE pre-training or pre-trained embeddings) to achieve reasonable performance. Representation Modeling Methods (RMMs), which avoid direct word probability modeling, have been largely unexplored due to poor performance, attributed by the authors to "invalid sampling" and "unhealthy gradients." This paper aims to revive RMMs by tackling these fundamental issues.

Methods

The paper proposes two key techniques to enable effective representation modeling in GANs:

  • Dropout Sampling: Instead of injecting random noise or sampling from word probabilities, this method masks dimensions of the generator's input representations. This introduces a non-negligible random factor, improving sample diversity and quality by mitigating mode collapse.
  • Fully Normalized LSTM: To address the issue of unhealthy gradients in deep sequence models, the authors introduce a variant of LSTM that incorporates layer normalization not just on the input but also within the hidden state calculation. Theoretical analysis and empirical results show this helps alleviate gradient vanishing and provides healthier gradients for training.

InitialGAN itself is a GAN composed of three components: an aligner (Transformer-based encoder) that maps words to representations, a generator that models these representations to produce new sequences, and a discriminator that distinguishes between real and generated representations. Crucially, all parameters in InitialGAN are randomly initialized, eliminating the need for pre-training.

Datasets

  • COCO Image Caption Dataset: Used for training and evaluation.
  • EMNLP 2017 News Dataset: Used for training and evaluation.

A new evaluation metric, Least Coverage Rate (LCR), is introduced to better assess the quality of generated samples, particularly for identifying mode collapse and out-of-distribution samples, as existing metrics like FED were found to be less sensitive to sample quality changes.

Key Insights and Ideas

  • Representation Modeling is a Promising Direction: The paper argues that RMMs, despite past failures, are a viable path for language GANs, especially when the core problems of sampling and gradients are addressed.
  • Pre-training is Not Always Necessary: InitialGAN demonstrates that a well-designed GAN architecture can achieve competitive or superior performance to MLE without any pre-training, challenging the conventional wisdom in language GAN research.
  • Dropout Sampling for Diversity: Masking representation dimensions is an effective way to inject randomness and improve sample diversity in RMMs, directly combating mode collapse.
  • Layer Normalization's Deeper Role: The authors provide theoretical and empirical evidence that layer normalization, when applied strategically within LSTM, can actively augment gradients, not just stabilize them.
  • LCR as a Sensitive Metric: The newly proposed LCR metric proves more effective than FED in distinguishing subtle differences in sample quality and distribution coverage between models.

Key Results

  • InitialGAN outperforms both MLE and other state-of-the-art language GANs (SeqGAN, RankGAN, MaliGAN, LeakGAN, ScratchGAN, RelGAN) on both COCO and EMNLP datasets, as measured by Inverse BLEU and the proposed LCR metric.
  • Crucially, InitialGAN achieves this superior performance without any pre-training, a feat not accomplished by any previous language GAN.
  • Ablation studies confirm the significant contributions of both dropout sampling and the fully normalized LSTM to InitialGAN's performance.
  • The paper shows that existing models like ScratchGAN, which rely on pre-trained embeddings, lose their advantage when these embeddings are removed, highlighting InitialGAN's unique ability to learn from random initialization.

Limitations and Future Work

  • Training Speed: The paper acknowledges that GANs, in general, have limitations in training speed due to their sequential nature, which prevents parallel computation structures like Transformers from being fully utilized. This remains a key challenge for applying language GANs to large datasets.
  • Mode Collapse: While InitialGAN mitigates mode collapse, the paper notes that further work might be needed to completely relieve it, as indicated by coverage rate analysis.
  • Further Exploration of RMMs: The success of InitialGAN suggests that RMMs are a fertile area for future research in language GANs.

Summary updated

Abstract

Text generative models trained via maximum likelihood estimation (MLE) suffer from the notorious exposure bias problem, and generative adversarial networks (GANs) are shown to have potential to tackle this problem. The existing language GANs adopt estimators, such as REINFORCE or continuous relaxations to model word probabilities. The inherent limitations of such estimators lead current models to rely on pretraining techniques (MLE pretraining or pretrained embeddings). Representation modeling methods (RMMs), which are free from those limitations, however, are seldomly explored because of their poor performance in previous attempts. Our analyses reveal that invalid sampling methods and unhealthy gradients are the main contributors to such unsatisfactory performance. In this work, we present two techniques to tackle these problems: dropout sampling and fully normalized long short-term memory network (LSTM). Based on these two techniques, we propose InitialGAN whose parameters are randomly initialized in full. Besides, we introduce a new evaluation metric, least coverage rate (LCR), to better evaluate the quality of generated samples. The experimental results demonstrate that the InitialGAN outperforms both MLE and other compared models. To the best of our knowledge, it is the first time a language GAN can outperform MLE without using any pretraining techniques.
BibTeX
@article{ren2024initialgan,
  author = {Da Ren and Qing Li},
  title = {{InitialGAN: A Language GAN With Completely Random Initialization}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2024},
  doi = {10.1109/tnnls.2023.3315778},
}
Figure