Abstract
Designed to learn long-range interactions on sequential data, transformers continue to show state-of-the-art results on a wide variety of tasks. In contrast to CNNs, they contain no inductive bias that prioritizes local interactions. This makes them expressive, but also computationally infeasible for long sequences, such as high-resolution images. We demonstrate how combining the effectiveness of the inductive bias of CNNs with the expressivity of transformers enables them to model and thereby synthesize high-resolution images. We show how to (i) use CNNs to learn a context-rich vocabulary of image constituents, and in turn (ii) utilize transformers to efficiently model their composition within high-resolution images. Our approach is readily applied to conditional synthesis tasks, where both non-spatial information, such as object classes, and spatial information, such as segmentations, can control the generated image. In particular, we present the first results on semantically-guided synthesis of megapixel images with transformers and obtain the state of the art among autoregressive models on class-conditional ImageNet.
Introduction
Transformers have revolutionized natural language processing by excelling at modeling long-range interactions. However, applying them to high-resolution imagery is notoriously difficult due to computational complexity that scales quadratically with sequence length. While a sentence might contain 50 words, a megapixel image contains over a million pixels, making standard self-attention mechanisms computationally infeasible. Previous attempts at "Image Transformers" were often restricted to low resolutions (e.g., $32 \times 32$) or local neighborhoods, sacrificing the global connectivity that makes Transformers powerful. This paper addresses the challenge of providing a Transformer with a holistic view of high-resolution images without hitting memory constraints.
Methodology: The VQGAN Approach
The proposed architecture utilizes a two-stage approach to bridge the gap between local efficiency and global expressivity.
Stage 1: Learning a Perceptually Rich Codebook
Instead of operating on raw pixels, the model learns a vocabulary of visual "parts." This is achieved through VQGAN, a CNN-based Vector Quantized Variational Autoencoder enhanced with adversarial and perceptual losses. This configuration allows for significant compression (e.g., reducing a $256 \times 256$ image to a $16 \times 16$ grid of indices) while ensuring reconstructions remain sharp and realistic rather than blurry.
Stage 2: Latent Transformer Modeling
Once the image is represented as a sequence of discrete codebook indices, a standard GPT-style Transformer is trained to predict the next index in the sequence (autoregressive modeling). Because the sequence length is drastically reduced, the Transformer can efficiently learn the global composition and semantic structure of the image.
For megapixel-scale synthesis, the authors employ a sliding window approach during inference. This allows the model to generate images of arbitrary size by attending to a local crop of previously generated visual parts, maintaining consistency across large spatial areas.
Experimental Results
The model was evaluated across diverse tasks, including class-conditional synthesis on ImageNet, high-resolution face generation (FacesHQ), and semantic-to-image synthesis (ADE20K, COCO-Stuff).
- Megapixel Synthesis: The model successfully generated images up to $1280 \times 460$ pixels, a milestone for autoregressive Transformer models.
- State-of-the-Art Performance: The approach achieved top-tier results among autoregressive models for class-conditional ImageNet synthesis, outperforming VQ-VAE-2.
- Efficiency and Versatility: Compared to pixel-based models like ImageGPT, this method is significantly faster and more memory-efficient. It also demonstrates flexibility across various conditioning inputs, such as depth maps, sketches, and pose skeletons.
Discussion and Key Insights
The core breakthrough of this work lies in the synergy between CNNs and Transformers. CNNs provide a strong inductive bias for local structure and spatial invariance, making them ideal for encoding low-level textures into "context-rich" parts. Transformers, lacking this bias, are then free to model the high-level, long-range relationships between these parts.
By using a GAN-based codebook, the model can effectively ignore stochastic pixel-level details that do not contribute to the overall semantic meaning, allowing the Transformer to focus on the structural arrangement of the scene. While sampling remains sequential and inherently slower than one-shot GAN generation, the ability to scale to megapixel resolutions represents a significant advancement in generative modeling.
Summary updated