Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers

Yanhong Zeng, Huan Yang, Hongyang Chao, Jianbo Wang, Jianlong Fu

Abstract

We present a new perspective of achieving image synthesis by viewing this task as a visual token generation problem. Different from existing paradigms that directly synthesize a full image from a single input (e.g., a latent code), the new formulation enables a flexible local manipulation for different image regions, which makes it possible to learn content-aware and fine-grained style control for image synthesis. Specifically, it takes as input a sequence of latent tokens to predict the visual tokens for synthesizing an image. Under this perspective, we propose a token-based generator (i.e., TokenGAN). Particularly, the TokenGAN inputs two semantically different visual tokens, i.e., the learned constant content tokens and the style tokens from the latent space. Given a sequence of style tokens, the TokenGAN is able to control the image synthesis by assigning the styles to the content tokens by attention mechanism with a Transformer. We conduct extensive experiments and show that the proposed TokenGAN has achieved state-of-the-art results on several widely-used image synthesis benchmarks, including FFHQ and LSUN CHURCH with different resolutions. In particular, the generator is able to synthesize high-fidelity images with 1024x1024 size, dispensing with convolutions entirely.

Introduction

For several years, the "Style-based" generator, exemplified by StyleGAN, has defined the state-of-the-art in image synthesis. These models typically rely on Adaptive Instance Normalization (AdaIN) to inject style information into the generation process. However, a fundamental limitation of AdaIN is its content-independence. By normalizing feature maps, AdaIN effectively "washes away" local information before applying a single, global style value. This entanglement makes it difficult to apply distinct styles to different image regions—such as changing eye color without affecting skin tone—often resulting in artifacts and limited fine-grained control.

Comparison of generative paradigms: (a) Standard GAN, (b) Style-based GAN, and (c) the proposed Token-based Generator (TokenGAN).

Methods: The Token-based Generator

To address these limitations, we propose TokenGAN, which reimagines image synthesis as a sequence-to-sequence visual token generation task. By replacing the traditional convolutional backbone with a Transformer-based architecture, TokenGAN achieves superior local control and detail reconstruction.

Architectural Innovation

Unlike previous models that synthesize a full image from a single latent code, TokenGAN operates on sequences of tokens. The core mechanism involves a Transformer that modulates content through cross-attention.

The TokenGAN architecture details the interaction between learned content tokens and style tokens derived from the latent space.

Key Components

  1. Visual Tokens: The image is represented as a sequence of patches or "visual tokens."
  2. Content Tokens: These are learned, constant inputs that serve as a structural "template" for the image.
  3. Style Tokens: The input latent code ($z$) is mapped via a mapping network ($M$) to a sequence of style tokens $\{s_1, ..., s_n\}$. This multi-token representation allows for more expressive stylistic attributes compared to a single vector.
  4. Transformer-based Style Modulation: Instead of AdaIN, TokenGAN uses cross-attention. Content tokens act as "Queries," while style tokens act as "Keys" and "Values." This allows the generator to dynamically attend to specific style tokens for different image regions.
  5. Convolution-Free Synthesis: The model can synthesize high-fidelity 1024 × 1024 images using only Transformers, leveraging their ability to model long-range dependencies without traditional convolutions.

Experimental Setup

TokenGAN was evaluated on high-resolution benchmarks:

  • FFHQ (Flickr-Faces-HQ): 70,000 high-quality human face images at 256×256 and 1024×1024 resolutions.
  • LSUN CHURCH: A dataset of complex architectural scenes to test the model's performance on structured, non-face environments.

Results and Insights

Performance Benchmarks

TokenGAN demonstrates superior visual quality and reconstruction accuracy compared to StyleGAN2:

  • Visual Quality (FID): On FFHQ-256, TokenGAN achieved an FID of 5.41 (vs. StyleGAN2's 6.09). At 1024×1024, it remained highly competitive with an FID of 5.21.
  • Reconstruction Accuracy: In image inversion tasks, TokenGAN significantly outperformed StyleGAN2, reducing Mean Average Error (MAE) from 16.45 to 13.43 and improving LPIPS from 0.1539 to 0.1238.
  • Distribution Coverage: Higher recall scores across benchmarks suggest the model better captures the full variety of the training data.

Unsupervised Semantic Discovery

A key insight is that the attention mechanism naturally learns semantic regions. Without manual labels, the model learns to associate specific style tokens with features like "eyes," "nose," or "hair." This allows for "local" edits—such as lengthening hair—without the "leakage" common in convolutional models.

Conclusion and Future Work

TokenGAN demonstrates that viewing image synthesis as token generation offers a powerful alternative to convolutional paradigms. While the model achieves state-of-the-art results, we observed that performance peaks at a specific number of style tokens (32), suggesting a trade-off between flexibility and convergence. Future work will focus on optimizing the computational efficiency of these Transformer-based generators and extending the approach to video and complex style transfer tasks.

Summary updated

Abstract

We present a new perspective of achieving image synthesis by viewing this task as a visual token generation problem. Different from existing paradigms that directly synthesize a full image from a single input (e.g., a latent code), the new formulation enables a flexible local manipulation for different image regions, which makes it possible to learn content-aware and fine-grained style control for image synthesis. Specifically, it takes as input a sequence of latent tokens to predict the visual tokens for synthesizing an image. Under this perspective, we propose a token-based generator (i.e.,TokenGAN). Particularly, the TokenGAN inputs two semantically different visual tokens, i.e., the learned constant content tokens and the style tokens from the latent space. Given a sequence of style tokens, the TokenGAN is able to control the image synthesis by assigning the styles to the content tokens by attention mechanism with a Transformer. We conduct extensive experiments and show that the proposed TokenGAN has achieved state-of-the-art results on several widely-used image synthesis benchmarks, including FFHQ and LSUN CHURCH with different resolutions. In particular, the generator is able to synthesize high-fidelity images with 1024x1024 size, dispensing with convolutions entirely.
BibTeX
@inproceedings{zeng2021improving,
  author = {Yanhong Zeng and Huan Yang and Hongyang Chao and Jianbo Wang and Jianlong Fu},
  title = {{Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=lGoKo9WS2A_},
}
Figure