Style-Guided Inference of Transformer for High-Resolution Image Synthesis

Jonghwa Yim, Minjae KimWACV 2023

TL;DR

This paper introduces a novel method for controlling the style of high-resolution images generated by autoregressive transformers. By treating a style image as a conditional input, the model can guide the transformer's inference process to produce outputs that match the desired style without requiring retraining, offering a more flexible and efficient approach to style-guided image synthesis.

What Problem Does This Paper Address?

Generating high-resolution, photorealistic images is a challenging task in computer vision. While autoregressive transformers, especially when combined with vector-quantized latent representations, have shown great promise in producing realistic images, a significant limitation arises during the sampling process. The inherent randomness in sampling discrete latent codes from the transformer's probability distribution leads to unpredictable outputs. This often necessitates generating numerous samples to acquire a desired result, which is computationally expensive and inefficient, particularly for high-resolution images. Existing methods for style transfer often require extensive retraining or large datasets, making them less practical for scenarios where only a few style examples are available. This paper aims to bridge this gap by enabling style-guided image generation with pre-trained transformers, offering a way to control the output's style without the need for additional training.

Methods

The core of this work builds upon the "Taming Transformers" [10] approach, which uses an autoregressive transformer to predict a sequence of quantized latent codes that are then decoded into an image. The authors' key innovation lies in style-guided inference. Instead of directly sampling from the transformer's learned prior distribution, they introduce a mechanism to condition this sampling process on a given style image.

Here's a breakdown of their approach:

  • Style Likelihood: They formulate a "style likelihood" ($P(z_u|T)$) that captures the relationship between the latent codes of the style image ($z_u$) and the transformer's learned prior ($P(T)$). This likelihood is derived from the probability distributions of latent codes extracted from the style image and the training dataset.
  • Posterior Formulation: The transformer's inference is guided by a modified posterior distribution that incorporates this style likelihood. This effectively re-balances the prior distribution to favor latent codes that align with the input style.
  • No Retraining: Crucially, this style guidance is achieved without retraining the pre-trained transformer. The method leverages the existing learned representations and guides the inference process through a probabilistic manipulation.
  • Semantic Guidance (Optional): For tasks like landscape synthesis, they can further enhance style transfer by using semantic segmentation maps. This allows for applying different styles to different semantic regions (e.g., sky, mountains) within the generated image, leading to more precise and semantically aware stylization.

Datasets

  • Landscape Synthesis: The authors collected approximately 80,000 landscape images from various sources (Flickr, Pixabay, Google Search) for training. Semantic labels were generated using DeepLabv3 with a ResNeSt-101 backbone.
  • Face Synthesis: For unconditional face synthesis, they used the Flickr-Faces-HQ (FFHQ) dataset.
  • WikiArt Dataset [8]: Used for an ablation study on unconditional image synthesis to evaluate the impact of spatial information on style transfer.

Key Insights and Ideas

  • Decoupling Style and Generation: The method effectively decouples the style of the generated image from the underlying generative model's inherent biases. This allows for greater control over the aesthetic qualities of the output.
  • Probabilistic Style Control: By reformulating style transfer as a probabilistic guidance problem, the authors offer a more principled and flexible way to control image generation compared to traditional methods.
  • Leveraging Pre-trained Models: The ability to guide inference without retraining is a significant advantage, as it allows users to leverage powerful pre-trained generative models and adapt their outputs to new styles with minimal effort.
  • Semantic-Aware Stylization: The integration of semantic information allows for fine-grained control over style application, enabling more realistic and contextually appropriate stylization.

Key Results

  • Qualitative Results: The paper presents compelling qualitative results demonstrating the method's ability to transfer styles from single images or categorized sets of images to generated landscapes and faces. The generated images exhibit styles that are visually similar to the input style images.
  • Quantitative Evaluation (FFHQ): For unconditional face synthesis, the method achieved lower Frechet Inception Distance (FID) scores compared to random sampling from the baseline model and a state-of-the-art competitor [28], indicating higher quality and more diverse generated images. For example, for "Babies" and "Sunglasses" categories, their FID scores were 74.39 and 42.13 respectively, significantly outperforming random sampling (94.21 and 39.15).
  • Ablation Study (WikiArt): The ablation study on the WikiArt dataset showed that even without explicit spatial guidance, their style-guided inference method achieved lower FID scores than random sampling, though the improvement was less dramatic.

Limitations and Future Work

  • Domain Limitation: The method cannot generate samples that fall outside the distribution of the training dataset. For instance, if trained on real faces, it cannot generate "Sketches" style output.
  • Spatial Information Loss: Aggregating probability distributions over entire latent maps without considering spatial relationships can lead to limited results. The authors acknowledge this and suggest using semantic label maps or coordinates to mitigate this.
  • Future Work:
    • Developing a general conditional autoregressive module to support a wider range of arbitrary conditional inputs.
  • Exploring "space-aware" style-guided inference to better leverage the spatial layout and sequence of latent codes from style images for more accurate spatial application of style.

Summary updated

BibTeX
@inproceedings{yim2023style,
  author = {Jonghwa Yim and Minjae Kim},
  title = {{Style-Guided Inference of Transformer for High-Resolution Image Synthesis}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure