Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers

Yanhong Zeng, Huan Yang, Hongyang Chao, Jianbo Wang, Jianlong Fu

TL;DR

TokenGAN reimagines image synthesis as a visual token generation problem rather than a global pixel-mapping task. By replacing traditional convolutional layers with a Transformer-based generator, it achieves content-aware, fine-grained style control and produces high-fidelity 1024×1024 images that surpass StyleGAN2 in both quality and reconstruction accuracy.

The Problem with Style-Based Generators

State-of-the-art image synthesis has long been dominated by "style-based" generators (like StyleGAN). These models use a technique called Adaptive Instance Normalization (AdaIN) to apply a style code to an image. However, the authors identify two major flaws in this paradigm:

  1. Content-Independence: AdaIN "washes away" the original feature information through normalization before applying new styles. It doesn't "look" at what is in the image region before styling it.
  2. Global Entanglement: A single style code typically affects the entire feature map. This makes it difficult to change one part of an image (like hair length) without accidentally affecting another (like the background or face shape). This entanglement often leads to visible artifacts, such as the infamous "droplet" artifacts in early StyleGAN versions.

To address these issues, the authors propose moving away from global feature maps toward a token-based approach that allows for localized, content-aware manipulation.

Comparison of Generator Architectures

Methods: Introducing TokenGAN

The core innovation is TokenGAN, a generator that dispenses with convolutions entirely in favor of a Transformer architecture. As seen in the comparison above, TokenGAN shifts from generating monolithic feature maps to generating discrete visual tokens.

  • Visual Tokens: The image is treated as a sequence of patches (tokens).
  • Content vs. Style Tokens: The model takes two types of inputs. Content tokens are learned constants that represent the structural "canvas" of the image. Style tokens are projected from the latent space ($z$) via a mapping network.
  • Cross-Attention Mechanism: Instead of using AdaIN, TokenGAN uses a Transformer to model the relationship between these tokens. The content tokens act as "Queries," while the style tokens act as "Keys" and "Values." This allows the model to dynamically decide which style tokens should influence which parts of the image (e.g., "apply the 'blue' style token only to the 'eyes' content tokens").
  • Style Modulation: Unlike standard Transformers that use residual additions, TokenGAN uses the attention results to modulate (multiply) the content tokens, effectively "painting" the style onto the structure.
  • Skip-Generator Architecture: To handle high resolutions, the model generates images at progressively higher scales (from $4^2$ to $1024^2$) and sums them, similar to the architecture of StyleGAN2 but using tokens instead of pixels.

Datasets

The authors evaluated TokenGAN on two standard, high-stakes benchmarks for generative modeling:

  • FFHQ (Flickr-Faces-HQ): 70,000 high-quality human face images, tested at both $256 \times 256$ and $1024 \times 1024$ resolutions.
  • LSUN CHURCH: A dataset of complex architectural scenes used to test the model's ability to handle structured, non-face environments.

Key Insights

  • The Power of Localization: By treating the image as a sequence, the Transformer can attend to specific style tokens for specific regions. This leads to "unsupervised semantic discovery"—the model naturally learns to associate certain style tokens with eyes, hair, or backgrounds without being told what they are.
  • LayerNorm > InstanceNorm: The authors found that using LayerNorm instead of InstanceNorm preserves the relative magnitude of features. This prevents the "washing away" of information and eliminates the droplet artifacts common in convolutional GANs.
  • Convolution-Free High-Res: This paper is a significant proof-of-concept that Transformers can synthesize high-fidelity $1024 \times 1024$ images without a single convolutional layer in the generator, challenging the long-held assumption that CNNs are mandatory for spatial synthesis.

Key Results

TokenGAN achieves state-of-the-art (SOTA) performance across several metrics:

  • Visual Quality (FID): On FFHQ-256, TokenGAN achieved an FID of 5.41, significantly better than StyleGAN2's 6.09. It also showed improvements on LSUN Church (5.56 vs 5.88).
  • Image Inversion: This is where the model truly shines. When trying to reconstruct a real image from a latent code, TokenGAN achieved a Mean Average Error (MAE) of 13.43, compared to StyleGAN2’s 16.45. Qualitatively, TokenGAN is much better at capturing fine details like the exact color of a headscarf or specific facial expressions.
  • Localized Editing: The authors demonstrated that they could change "hair length" or "pose" with much less leakage into other parts of the image compared to previous methods.

Limitations and Future Work

  • Optimization Sensitivity: The authors noted that the number of style tokens matters; increasing the count to 64 actually decreased performance (FID dropped to 7.60), suggesting that too many tokens make the optimization landscape difficult for the Transformer to navigate.
  • Computational Cost: While the paper demonstrates 1024x1024 synthesis, Transformers generally have quadratic complexity relative to sequence length. The "skip-generator" helps mitigate this, but scaling to even higher resolutions or video remains a computational challenge.
  • Future Directions: The authors suggest extending this token-based approach to other tasks like image inpainting, animation generation, and more complex style transfer scenarios.

Summary updated

Abstract

We present a new perspective of achieving image synthesis by viewing this task as a visual token generation problem. Different from existing paradigms that directly synthesize a full image from a single input (e.g., a latent code), the new formulation enables a flexible local manipulation for different image regions, which makes it possible to learn content-aware and fine-grained style control for image synthesis. Specifically, it takes as input a sequence of latent tokens to predict the visual tokens for synthesizing an image. Under this perspective, we propose a token-based generator (i.e., TokenGAN). Particularly, the TokenGAN inputs two semantically different visual tokens, i.e., the learned constant content tokens and the style tokens from the latent space. Given a sequence of style tokens, the TokenGAN is able to control the image synthesis by assigning the styles to the content tokens by attention mechanism with a Transformer. We conduct extensive experiments and show that the proposed TokenGAN has achieved state-of-the-art results on several widely-used image synthesis benchmarks, including FFHQ and LSUN CHURCH with different resolutions. In particular, the generator is able to synthesize high-fidelity images with (1024x1024) size, dispensing with convolutions entirely.
BibTeX
@inproceedings{zeng2021improving,
  author = {Yanhong Zeng and Huan Yang and Hongyang Chao and Jianbo Wang and Jianlong Fu},
  title = {{Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=lGoKo9WS2A_},
}
Figure