Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?

Kevin Blin, Andrei Kucharavy

Abstract

While Transformers have become the standard for most Natural Language Processing (NLP) tasks, text-generating Generative Adversarial Networks (GANs) still predominantly rely on Recurrent Neural Networks (RNNs). This paper investigates the feasibility of a "drop-in" replacement of LSTM layers with Transformer layers in the DPGAN architecture. Our findings suggest that this simple substitution is insufficient, leading to poor pre-training performance and catastrophic mode collapse during adversarial tuning.

Introduction

The transition from Recurrent Neural Networks (RNNs) to Transformers has defined the modern era of NLP. Transformers offer significant advantages in scalability and long-range dependency modeling through their self-attention mechanism. However, the field of text-generating GANs has been slow to adopt this shift. Most state-of-the-art text GANs, such as the Diversity-Promoting GAN (DPGAN), still rely on LSTMs or GRUs. This study explores whether the benefits of Transformers can be easily harvested by performing a direct architectural swap.

Methodology

The researchers developed SADPGAN (Self-Attention DPGAN) by replacing the LSTM layers of the original DPGAN with Transformer layers. To optimize the integration, three experimental configurations were tested:

  1. Encoder-only: Utilizing only Transformer encoders, which proved unable to learn generative patterns.
  2. Encoder-Decoder with Empty Target: Providing the decoder with an empty vector to maximize generative freedom, which resulted in immediate collapse.
  3. Teacher Forcing and Autoregressive Sampling: Implementing a standard autoregressive setup where the decoder is trained with teacher forcing and evaluated via word-by-word sampling.

The models were evaluated on synthetic data and the MS COCO dataset using Negative Log-Likelihood for quality ($NLL_{gen}$) and diversity ($NLL_{div}$), alongside BLEU scores.

MLE pre-training results for early SADPGAN iterations, showing a failure to minimize NLL compared to the DPGAN baseline.

Results and Analysis

The results demonstrate a clear "drop-in failure." In the initial experiments, SADPGAN failed to show any learning progress during the Maximum Likelihood Estimation (MLE) pre-training phase, with NLL scores remaining stagnant.

Even in the most successful configuration (Experiment 3), the Transformer-based model struggled. While it achieved better $NLL_{gen}$ than previous iterations, it remained significantly behind the LSTM-based DPGAN. Furthermore, this marginal gain in quality was offset by a "massive cost to diversity," with $NLL_{div}$ scores increasing over time, indicating the model was becoming more repetitive.

MLE pre-training results for Experiment 3, demonstrating that even with teacher forcing, SADPGAN underperforms in quality and diversity.

The most critical failure occurred during the adversarial fine-tuning phase. While the original DPGAN improved its BLEU scores (e.g., $n=2$ BLEU rising from 0.73 to 0.845), SADPGAN suffered from catastrophic mode collapse. Qualitatively, the model's output degraded from coherent sentences like "a man riding a motorcycle down a street" to literal empty strings. This suggests that Transformers, which are notoriously data-hungry, overfit the training distribution during MLE and cannot adapt to the "moving target" of an adversarial critic.

Conclusion

This research highlights that Transformers are not a plug-and-play substitute for RNNs in the volatile environment of GAN training. The reliance on teacher forcing creates a training-inference mismatch that is exacerbated by the adversarial loop. The authors suggest that future work should incorporate specialized regularization, such as spectral normalization, and specific learning rate warm-up schedules. Ultimately, researchers should move away from "drop-in" replacements and instead focus on designing GAN architectures from the ground up to accommodate the unique training dynamics of self-attention mechanisms.

Summary updated

Abstract

In this paper we address the problem of fine-tuned text generation with a limited computational budget. For that, we use a well-performing text generative adversarial network (GAN) architecture - Diversity-Promoting GAN (DPGAN), and attempted a drop-in replacement of the LSTM layer with a self-attention-based Transformer layer in order to leverage their efficiency. The resulting Self-Attention DPGAN (SADPGAN) was evaluated for performance, quality and diversity of generated text and stability. Computational experiments suggested that a transformer architecture is unable to drop-in replace the LSTM layer, under-performing during the pre-training phase and undergoing a complete mode collapse during the GAN tuning phase. Our results suggest that the transformer architecture need to be adapted before it can be used as a replacement for RNNs in text-generating GANs.
BibTeX
@inproceedings{blin2021can,
  author = {Kevin Blin and Andrei Kucharavy},
  title = {{Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?}},
  booktitle = {RANLP},
  year = {2021},
  url = {https://openreview.net/forum?id=JDLOdGIPwH},
}
Figure