Generative Adversarial Transformers

Drew A. Hudson, Larry Zitnick

Introduction

The GANsformer is a novel architecture that marries the global reasoning capabilities of Transformers with the high-fidelity synthesis of Generative Adversarial Networks (GANs). By replacing standard quadratic self-attention with a more efficient bipartite attention mechanism, it achieves state-of-the-art results in generating complex, multi-object scenes while remaining computationally lean enough for high-resolution images.

The Problem: Local Bias vs. Quadratic Complexity

For the past decade, Convolutional Neural Networks (CNNs) have been the workhorse of computer vision. However, CNNs have a fundamental "local" bias—they process images through small receptive fields, making it difficult for them to coordinate distant parts of a scene (like ensuring the lighting on a chair matches the window across the room).

Transformers offer a solution with their "global" attention, but they come with a heavy price: quadratic complexity. In a standard Transformer, every pixel attends to every other pixel ($O(n^2)$). For a high-resolution image, this memory and computation requirement quickly becomes prohibitive. The authors seek to bridge this gap, creating a model that can handle long-range dependencies without the massive computational overhead, specifically for the task of generating realistic images.

Methods: The Bipartite Transformer

The core innovation of the GANsformer is the Bipartite Transformer. Instead of letting every pixel talk to every other pixel, the authors introduce a small set of "latent variables" (aggregator nodes) that act as a communication bottleneck.

  • Bipartite Structure: The image features ($n$ pixels) interact with a compact set of latents ($m$ variables, typically 8–32). This reduces complexity from $O(n^2)$ to $O(mn)$, which is effectively linear relative to the image size.
  • Simplex vs. Duplex Attention: The paper introduces two modes. Simplex attention allows information to flow one way (usually from latents to pixels). Duplex attention enables bidirectional flow, mimicking the "top-down" and "bottom-up" processing found in human perception.
  • Multiplicative Integration: Borrowing a successful trick from StyleGAN, the model doesn't just add the attended information to the features; it uses it to modulate the scale and bias (gain and offset) of the features. This allows for much finer control over the "style" of specific regions.
Comparison of standard self-attention with the proposed Bipartite Transformer's Simplex and Duplex attention mechanisms.

Datasets

The authors evaluated the GANsformer on four diverse datasets to test its robustness across different levels of complexity:

  1. CLEVR: A synthetic dataset of geometric objects, perfect for testing how well a model understands composition and object relationships.
  2. LSUN-Bedroom: A challenging real-world dataset of indoor scenes with high layout diversity.
  3. Cityscapes: Urban street scenes, testing the model's ability to handle complex, structured outdoor environments.
  4. FFHQ: High-quality human faces, used as a baseline for datasets where the layout is relatively consistent.

Key Insights and Ideas

  • Latents as Object "Owners": One of the most striking findings is that the latent variables naturally specialize. One latent might learn to "control" the bed in a room, while another handles the windows. This emerges automatically during training without explicit supervision.
  • The Power of the Bottleneck: By forcing communication through a small set of latents, the model learns a more "compositional" representation of the world. It treats a scene as a collection of interacting entities rather than just a grid of pixels.
  • Data Efficiency: Because the inductive bias (the bipartite structure) matches the inherent compositionality of real-world scenes, the GANsformer is remarkably data-efficient. It can produce high-quality results even when trained on significantly smaller datasets compared to standard GANs.
Visualizing attention maps reveals how different latent variables naturally specialize in specific semantic regions across various datasets.

Key Results

The GANsformer outperformed several strong baselines, including StyleGAN2 and SAGAN, across multiple metrics:

  • Image Quality (FID): On the CLEVR dataset, the GANsformer improved the FID score (lower is better) from 16.05 (StyleGAN2) to 9.17. On the Bedroom dataset, it nearly halved the FID from 11.53 to 6.51.
  • Training Speed: The model reached high-quality synthesis on CLEVR roughly 3x faster than the next best approach.
  • Disentanglement: Using the DCI (Disentanglement, Completeness, Informativeness) metric, the authors showed that the GANsformer’s latent space is significantly more organized and interpretable than that of a standard GAN.

Limitations and Future Work

While the GANsformer excels at compositional scenes, the gains on highly structured, low-diversity datasets like FFHQ (faces) were more modest. This suggests that the bipartite attention is most beneficial when there is significant spatial variety to manage.

The authors note that while they focused on image synthesis, the Bipartite Transformer is a general-purpose architecture. Future work could involve applying this efficient attention mechanism to other domains, such as video generation, image recognition, or even large-scale language modeling where long-range context is currently a bottleneck.

Summary updated

Abstract

We introduce the GANsformer, a novel and efficient type of transformer, and explore it for the task of visual generative modeling. The network employs a bipartite structure that enables long-range ...
BibTeX
@inproceedings{hudson2021generative,
  author = {Drew A. Hudson and Larry Zitnick},
  title = {{Generative Adversarial Transformers}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=1i-b5uRMIOF},
}
Figure