Abstract
While Transformers have become the standard for most Natural Language Processing (NLP) tasks, text-generating Generative Adversarial Networks (GANs) still predominantly rely on Recurrent Neural Networks (RNNs). This paper investigates the feasibility of a "drop-in" replacement of LSTM layers with Transformer layers in the DPGAN architecture. Our findings suggest that this simple substitution is insufficient, leading to poor pre-training performance and catastrophic mode collapse during adversarial tuning.
Introduction
The transition from Recurrent Neural Networks (RNNs) to Transformers has defined the modern era of NLP. Transformers offer significant advantages in scalability and long-range dependency modeling through their self-attention mechanism. However, the field of text-generating GANs has been slow to adopt this shift. Most state-of-the-art text GANs, such as the Diversity-Promoting GAN (DPGAN), still rely on LSTMs or GRUs. This study explores whether the benefits of Transformers can be easily harvested by performing a direct architectural swap.
Methodology
The researchers developed SADPGAN (Self-Attention DPGAN) by replacing the LSTM layers of the original DPGAN with Transformer layers. To optimize the integration, three experimental configurations were tested:
- Encoder-only: Utilizing only Transformer encoders, which proved unable to learn generative patterns.
- Encoder-Decoder with Empty Target: Providing the decoder with an empty vector to maximize generative freedom, which resulted in immediate collapse.
- Teacher Forcing and Autoregressive Sampling: Implementing a standard autoregressive setup where the decoder is trained with teacher forcing and evaluated via word-by-word sampling.
The models were evaluated on synthetic data and the MS COCO dataset using Negative Log-Likelihood for quality ($NLL_{gen}$) and diversity ($NLL_{div}$), alongside BLEU scores.
Results and Analysis
The results demonstrate a clear "drop-in failure." In the initial experiments, SADPGAN failed to show any learning progress during the Maximum Likelihood Estimation (MLE) pre-training phase, with NLL scores remaining stagnant.
Even in the most successful configuration (Experiment 3), the Transformer-based model struggled. While it achieved better $NLL_{gen}$ than previous iterations, it remained significantly behind the LSTM-based DPGAN. Furthermore, this marginal gain in quality was offset by a "massive cost to diversity," with $NLL_{div}$ scores increasing over time, indicating the model was becoming more repetitive.
The most critical failure occurred during the adversarial fine-tuning phase. While the original DPGAN improved its BLEU scores (e.g., $n=2$ BLEU rising from 0.73 to 0.845), SADPGAN suffered from catastrophic mode collapse. Qualitatively, the model's output degraded from coherent sentences like "a man riding a motorcycle down a street" to literal empty strings. This suggests that Transformers, which are notoriously data-hungry, overfit the training distribution during MLE and cannot adapt to the "moving target" of an adversarial critic.
Conclusion
This research highlights that Transformers are not a plug-and-play substitute for RNNs in the volatile environment of GAN training. The reliance on teacher forcing creates a training-inference mismatch that is exacerbated by the adversarial loop. The authors suggest that future work should incorporate specialized regularization, such as spectral normalization, and specific learning rate warm-up schedules. Ultimately, researchers should move away from "drop-in" replacements and instead focus on designing GAN architectures from the ground up to accommodate the unique training dynamics of self-attention mechanisms.
Summary updated