Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?

Kevin Blin, Andrei Kucharavy

Abstract

In this paper, we address the problem of fine-tuned text generation with a limited computational budget. We utilize a well-performing text generative adversarial network (GAN) architecture—Diversity-Promoting GAN (DPGAN)—and attempt a drop-in replacement of the LSTM layer with a self-attention-based Transformer layer to leverage its efficiency. The resulting Self-Attention DPGAN (SADPGAN) was evaluated for performance, quality, diversity of generated text, and stability. Computational experiments suggested that a Transformer architecture is unable to drop-in replace the LSTM layer, under-performing during the pre-training phase and undergoing a complete mode collapse during the GAN tuning phase. Our results suggest that the Transformer architecture needs to be adapted before it can be used as a replacement for RNNs in text-generating GANs.

Introduction

While Transformers have largely replaced Recurrent Neural Networks (RNNs) in most NLP domains, text-based Generative Adversarial Networks (GANs) have remained a stronghold for LSTMs. This study explores whether the Transformer's efficiency and long-range dependency modeling can be easily ported into the GAN framework as a "drop-in" replacement for the traditional LSTM generator.

The Challenge of Text GANs

Text generation via GANs is inherently difficult due to the discrete nature of text, which prevents direct gradient-based optimization. Most architectures rely on a two-stage process:

  1. Maximum Likelihood Estimation (MLE) Pre-training: The model is trained as a standard language model to predict the next word.
  2. Adversarial Fine-tuning: A Generator attempts to fool a Discriminator into accepting its generated sequences as real data.

Methodology

The researchers used the Diversity Promoting GAN (DPGAN) as a baseline, creating a variant called SADPGAN (Self-Attention DPGAN). They tested several configurations, including encoder-only and encoder-decoder setups with teacher forcing. To maintain a fair comparison, both models were kept relatively small, with hidden dimensions of 32 and 2 layers.

Experimental Results

Pre-training Performance

The first significant hurdle for the SADPGAN was the MLE pre-training phase. Despite the Transformer's reputation for language modeling, the drop-in configuration struggled to minimize loss compared to the LSTM baseline.

MLE pre-training loss (NLL_gen) for DPGAN vs SADPGAN. The Transformer-based model (red) fails to minimize loss compared to the LSTM baseline (blue).

As shown in Figure 1, the generation loss ($NLL_{gen}$) for the SADPGAN remained nearly static, while the DPGAN showed consistent improvement. This trend extended to diversity metrics as well; while the DPGAN successfully lowered its diversity loss ($NLL_{div}$), the SADPGAN often saw this metric increase or plateau at a high level.

Comparison of generation loss (NLL_gen) and diversity loss (NLL_div) during pre-training. SADPGAN shows significantly higher losses and poor diversity trends.

Adversarial Failure and Mode Collapse

The transition to adversarial training proved even more problematic. During the GAN tuning phase, the SADPGAN frequently suffered from total "mode collapse." In many instances, the model began producing exclusively empty sentences or repetitive, nonsensical word sequences to satisfy the discriminator. Quantitatively, the SADPGAN's BLEU scores often became nan during this phase, while the DPGAN achieved a BLEU-2 of 0.73.

Discussion and Conclusion

The failure of the Transformer as a drop-in replacement can be attributed to several factors:

  • Teacher Forcing Dependency: Transformers rely heavily on seeing the correct previous word during training. When this "crutch" is removed during the autoregressive sampling of the GAN phase, the model collapses.
  • Data Hunger and Overfitting: On the smaller datasets typical of GAN research (like MS COCO), the Transformer tends to overfit the training distribution rapidly, a problem amplified by the volatile adversarial loop.
  • Stability: LSTMs appear to offer a more stable sequential bias that is better suited for the current GAN training paradigms.

In conclusion, while Transformers are powerful, they cannot be used as a simple plug-and-play component in existing text GAN architectures. Future work must focus on stabilization techniques, such as spectral normalization or specialized learning rate schedulers, to successfully integrate self-attention into the adversarial text generation framework.

Summary updated

Abstract

In this paper we address the problem of fine-tuned text generation with a limited computational budget. For that, we use a well-performing text generative adversarial network (GAN) architecture - Diversity-Promoting GAN (DPGAN), and attempted a drop-in replacement of the LSTM layer with a self-attention-based Transformer layer in order to leverage their efficiency. The resulting Self-Attention DPGAN (SADPGAN) was evaluated for performance, quality and diversity of generated text and stability. Computational experiments suggested that a transformer architecture is unable to drop-in replace the LSTM layer, under-performing during the pre-training phase and undergoing a complete mode collapse during the GAN tuning phase. Our results suggest that the transformer architecture need to be adapted before it can be used as a replacement for RNNs in text-generating GANs.
BibTeX
@inproceedings{blin2021can,
  author = {Kevin Blin and Andrei Kucharavy},
  title = {{Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?}},
  booktitle = {RANLP},
  year = {2021},
  url = {https://openreview.net/forum?id=JDLOdGIPwH},
}
Figure