Abstract
Designed to learn long-range interactions on sequential data, transformers continue to show state-of-the-art results on a wide variety of tasks. In contrast to CNNs, they contain no inductive bias that prioritizes local interactions. This makes them expressive, but also computationally infeasible for long sequences, such as high-resolution images. We demonstrate how combining the effectiveness of the inductive bias of CNNs with the expressivity of transformers enables them to model and thereby synthesize high-resolution images. We show how to (i) use CNNs to learn a context-rich vocabulary of image constituents, and in turn (ii) utilize transformers to efficiently model their composition within high-resolution images. Our approach is readily applied to conditional synthesis tasks, where both non-spatial information, such as object classes, and spatial information, such as segmentations, can control the generated image. In particular, we present the first results on semantically-guided synthesis of megapixel images with transformers.
Introduction: The Resolution Bottleneck
Transformers have revolutionized natural language processing by capturing long-range interactions. However, applying them to high-resolution images is computationally prohibitive because their self-attention mechanism scales quadratically with sequence length. While a sentence may contain dozens of tokens, a megapixel image contains millions of pixels.
Conversely, Convolutional Neural Networks (CNNs) are efficient due to their inductive bias—the assumption of local pixel correlation. While excellent for capturing textures, CNNs often struggle with global structural consistency, such as maintaining symmetry in complex scenes. This work proposes a hybrid approach: using CNNs to handle local visual "parts" and Transformers to manage global "logic."
Methodology: Two-Stage Synthesis
The proposed architecture, VQGAN, operates in two distinct stages to bridge the gap between local efficiency and global expressivity.
Stage 1: Learning a Context-Rich Vocabulary
The first stage involves training a Vector Quantized GAN (VQGAN). Unlike standard VQ-VAEs, this model incorporates a patch-based discriminator and perceptual loss. This combination forces the encoder to compress images into a grid of discrete codebook entries that represent perceptually meaningful visual constituents rather than raw pixels. This "context-rich" vocabulary allows for high compression ratios (e.g., reducing a $256 \times 256$ image to a $16 \times 16$ grid) without sacrificing visual crispness. By adding a GAN objective, the authors "tame" the Transformer, allowing it to focus on high-level arrangement rather than modeling every tiny pixel noise.
Stage 2: Global Composition via Transformers
With the image represented as a compact sequence of codebook indices, a Transformer is trained to model their distribution autoregressively. By operating on the compressed latent space, the Transformer avoids the quadratic explosion of pixel-level modeling. To extend synthesis to megapixel resolutions, the authors employ a sliding-window attention mechanism, allowing the model to generate large-scale images by attending to a local neighborhood of previously generated tokens.
Key Results and Applications
The hybrid approach enables high-fidelity synthesis across various domains, including faces (FacesHQ), architecture (LSUN), and landscapes (S-FLCKR).
- Megapixel Synthesis: The model achieves the first results for semantically-guided synthesis of megapixel images using Transformers, maintaining both local texture and global coherence.
- Quantitative Performance: VQGAN outperforms state-of-the-art models like PixelSNAIL and VQ-VAE-2 in Fréchet Inception Distance (FID) and Negative Log-Likelihood (NLL). Sampling is roughly 14x faster than Transformers operating directly on pixels.
- Unified Conditional Synthesis: The architecture is highly flexible. By prepending conditioning information—such as class labels, depth maps, or segmentation masks—to the token sequence, the same model can perform diverse tasks like pose-guided or layout-to-image synthesis without task-specific architectural changes.
Discussion and Limitations
While VQGAN significantly "tames" the Transformer for image synthesis, some challenges remain. The autoregressive sampling process, though faster than pixel-level models, is still slower than one-shot GAN generators. Additionally, the quality of the final output is fundamentally limited by the Stage 1 codebook; the Transformer cannot generate visual features that the VQGAN encoder failed to capture. Future work may explore non-autoregressive modeling or further improvements to codebook efficiency to mitigate sliding window artifacts.
Summary updated