A Representation Modeling Based Language GAN with Completely Random Initialization

Da Ren, Qing Li

Introduction

Training generative models for discrete text data has long been a challenge in natural language processing. While Maximum Likelihood Estimation (MLE) is the standard approach, it suffers from exposure bias, where the model's reliance on ground-truth tokens during training leads to error accumulation during autonomous inference. Generative Adversarial Networks (GANs) offer a potential solution by aligning training and inference environments, but the non-differentiable nature of discrete tokens has historically necessitated complex workarounds like reinforcement learning (REINFORCE) or Gumbel-Softmax. These methods, known as Probability Modeling Methods (PMMs), are often unstable and require extensive MLE pre-training to function.

InitialGAN introduces a paradigm shift by utilizing Representation Modeling Methods (RMMs). By mapping words to continuous vectors and employing a novel "fully normalized" LSTM architecture, InitialGAN becomes the first language GAN capable of outperforming MLE baselines starting from completely random initialization, without any pre-training or pre-trained embeddings.

Proposed Methodology

InitialGAN addresses the two primary failures of previous RMMs: invalid sampling (mode collapse) and unhealthy gradients (vanishing gradients in deep recurrent structures).

Architecture of InitialGAN showing the Aligner and Generator modules.

1. The Aligner

The Aligner component, built on a Variational Autoencoder (VAE) and Transformer encoder, maps discrete words into a continuous distribution. To ensure the representation space is used efficiently, the authors implement a modified loss function that prevents high-frequency words from dominating the vector space, thereby preserving the representability of rare tokens.

2. Dropout Sampling

To combat mode collapse—where a generator ignores random noise and produces repetitive outputs—InitialGAN utilizes dropout as a sampling mechanism. By masking specific dimensions of the input during the generation process, the model is forced to explore diverse combinations of values, significantly improving the variety and quality of the generated text.

3. Fully Normalized LSTM

The most significant theoretical advancement is the introduction of the Fully Normalized LSTM. The authors mathematically demonstrate that standard LSTMs tend to narrow gradients through their gating mechanisms. By applying Layer Normalization to the hidden state calculations, they introduce an "augmentation term" to the gradient. This ensures stable gradient flow even when many layers are stacked, which is critical for the adversarial training process.

Experimental Evaluation

The model was rigorously tested on two benchmarks: the COCO Image Caption Dataset (short sentences) and the EMNLP 2017 News Dataset (long sentences).

FED performance comparison across COCO and EMNLP datasets.

Key Results

  • Superior Performance: InitialGAN consistently outperformed both MLE and state-of-the-art GANs like SeqGAN, LeakGAN, and RelGAN. On the COCO dataset, InitialGAN achieved an Inverse BLEU score of 33.06, surpassing the MLE baseline of 32.03.
  • Zero Pre-training: Unlike competitors such as ScratchGAN (which requires pre-trained embeddings) or SeqGAN (which requires MLE "warm-up"), InitialGAN achieves these results starting from scratch.
  • Metric Sensitivity: The authors proposed a new metric, Least Coverage Rate (LCR), arguing that standard metrics like Frechet Embedding Distance (FED) are too coarse. LCR provides a more granular view of how well the generated distribution covers the real data distribution.

Conclusion and Future Work

InitialGAN proves that representation modeling is a viable and superior alternative to probability modeling for text GANs. By solving the stability and gradient issues inherent in deep recurrent adversarial networks, it removes the dependency on MLE pre-training. Future research will focus on optimizing the training speed of the autoregressive structure and further scaling the approach to massive, web-scale datasets to test the limits of its representation capabilities.

Summary updated

BibTeX
@article{ren2024representation,
  author = {Da Ren and Qing Li},
  title = {{A Representation Modeling Based Language GAN with Completely Random Initialization}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2024},
  doi = {10.1109/tnnls.2023.3315778},
}
Figure