Abstract: The recent explosive interest in transformers has suggested their potential to become powerful "universal" models for computer vision tasks. While most attempts focus on discriminative models, we explore transformers for generative adversarial networks (GANs). We present TransGAN, the first pilot study building a GAN completely free of convolutions. TransGAN consists of a memory-friendly transformer-based generator that progressively increases feature resolution and a multi-scale discriminator to capture both semantic contexts and low-level textures. To scale to high-resolution generation, we introduce grid self-attention to alleviate memory bottlenecks. We also develop a unique training recipe including data augmentation, modified normalization, and relative position encoding to mitigate instability. TransGAN achieves highly competitive performance, setting a new state-of-the-art Inception Score of 10.43 and FID of 18.28 on STL-10, outperforming StyleGAN-V2. On 256x256 generation tasks like CelebA-HQ, TransGAN produces diverse visual examples with high fidelity and impressive texture details.
Introduction
For years, Convolutional Neural Networks (CNNs) have dominated computer vision due to their inductive biases—locality and spatial invariance. However, these strengths limit their ability to capture long-range dependencies and global structures without deep, complex hierarchies. While Transformers have revolutionized NLP and image classification, applying them to GANs is notoriously difficult due to two major hurdles:
- Memory Explosion: Self-attention has a quadratic complexity $O(N^2)$ relative to the number of pixels. Generating high-resolution images pixel-by-pixel is computationally prohibitive.
- Training Instability: GANs are inherently difficult to stabilize. Transformers, lacking the helpful "shortcuts" of convolutional biases, are even more prone to training failure and mode collapse.
The TransGAN Architecture
To overcome these challenges, TransGAN utilizes two pure Transformer modules designed for generative tasks.
Memory-Friendly Generator
Instead of treating a high-resolution image as a single long sequence, the generator works in stages. It begins with a low-resolution feature map (e.g., 8x8) and iteratively upscales it using pixel shuffling or bicubic interpolation while simultaneously reducing the embedding dimension. This progressive approach keeps the sequence length manageable at each stage of the generation process.
Multi-Scale Discriminator
To ensure the model captures both fine textures and global coherence, the discriminator utilizes a multi-scale approach. It takes patches of varying sizes (e.g., $P, 2P, 4P$) as inputs at different stages, allowing it to evaluate the image at multiple levels of granularity.
Scaling and Training Techniques
Scaling to high resolutions like 256x256 requires further innovation to manage the memory wall of self-attention.
Grid Self-Attention
The authors introduce Grid Self-Attention, which partitions the feature map into non-overlapping grids. Attention is calculated only within these grids rather than across the entire image. This drastically reduces the memory footprint while still allowing the model to learn local structures that expand as the resolution increases across stages.
The Training Recipe
Because Transformers lack the inductive biases of CNNs, they are significantly more "data-hungry." The authors identified three essential techniques for stable training:
- Data Augmentation (DiffAug): Crucial for providing enough variety for the Transformer to learn spatial relationships from scratch.
- Modified Normalization: Standard LayerNorm is replaced with a token-wise scaling layer to prevent signal magnitudes from exploding during the adversarial training process.
- Relative Position Encoding: This helps the model understand the spatial relationship between patches more effectively than absolute coordinates, which is vital for maintaining global structure.
Experimental Results
TransGAN was evaluated on standard benchmarks including CIFAR-10, STL-10, CelebA, and LSUN-Church. The results demonstrate that pure Transformers can match or exceed established CNN-based models:
- STL-10: TransGAN set a new state-of-the-art with an Inception Score (IS) of 10.43 and an FID of 18.28.
- CIFAR-10: It achieved an IS of 9.02 and FID of 9.26, outperforming many popular CNN architectures.
- High-Resolution Synthesis: On CelebA-HQ (256x256), the model produced diverse, high-fidelity faces with sharp details, proving its scalability beyond low-resolution thumbnails.
Key Insights and Conclusion
Visualizing the training dynamics revealed a fascinating behavior: TransGAN learns the global layout (the "big picture") very early on, whereas CNN-based GANs often focus on local textures first. This suggests that Transformers prioritize global coherence, a direct benefit of the attention mechanism.
While TransGAN represents a breakthrough, challenges remain. There is still a slight FID gap compared to heavily engineered models like StyleGAN2 at very high resolutions, and the model remains more data-hungry than its convolutional counterparts. However, by proving that "two transformers can make one strong GAN," this work opens the door for a new generation of convolution-free generative models.
Summary updated