SketchInverter: Multi-Class Sketch-Based Image Generation via GAN Inversion

Zirui An, Jingbo Yu, Runtao Liu, Chuang Wang, Qian YuWACV 2023 · OpenReview

Abstract

This paper proposes the first GAN inversion-based method for multi-class sketch-based image generation (MC-SBIG). MC-SBIG is a challenging task that requires strong prior knowledge due to the significant domain gap between sketches and natural images. Existing learning-based approaches rely on a large-scale paired dataset to learn the mapping between these two image modalities. However, since the public paired sketch-photo data are scarce, it is struggling for learning-based methods to achieve satisfactory results. In this work, we introduce a new approach based on GAN inversion, which can utilize a powerful pretrained generator to facilitate image generation from a given sketch. Our GAN inversion-based method has two advantages:

  1. it can freely take advantage of the prior knowledge of a pretrained image generator;
  2. it allows the proposed model to focus on learning the mapping from a sketch to a low-dimension latent code, which is a much easier task than directly mapping to a high-dimension natural image. We also present a novel shape loss to improve generation quality further. Extensive experiments are conducted to show that our method can produce sketch-faithful and photo-realistic images and significantly outperform the baseline methods.

Introduction

Generating realistic photos from free-hand sketches, known as Sketch-Based Image Generation (SBIG), is notoriously difficult due to the inherent domain gap. Sketches are abstract, geometrically deformed, and lack color or texture, while natural photos are dense and realistic. Most existing methods struggle with two main issues: data scarcity and multi-class complexity. There are very few large-scale datasets containing "paired" data (a sketch and its exact corresponding photo). Furthermore, while models can be trained to generate specific categories like faces or shoes, generating hundreds of different object categories requires immense prior knowledge about real-world object appearances.

Existing learning-based models often produce blurry or distorted results because they attempt to learn the mapping from sketch pixels to photo pixels directly—a mathematically difficult task. SketchInverter addresses these challenges by shifting the strategy from direct image translation to GAN inversion.

Methodology

SketchInverter leverages a fixed, pretrained BigGAN (trained on ImageNet) which already possesses the knowledge to create high-quality images across 1,000 different classes. The task is redefined as finding the specific latent code ($z$) in BigGAN's memory that matches a given input sketch.

Conditional Sketch Encoder

The framework utilizes a novel encoder ($E$) that takes both a sketch and a class label as input. It maps these to a low-dimensional latent code. Mapping to a 128-D vector is significantly easier for a network than mapping directly to a high-resolution pixel grid.

Shape Loss and Synthetic Training

To ensure the generated image faithfully follows the sketch's geometry, the authors introduced a novel shape loss. They use a separate "photo-to-sketch" network ($S$) to transform the generated image back into a sketch, which is then compared to the original input. To overcome the lack of paired data, the team used BigGAN to generate a synthetic training set of 12,000 pairs by sampling random codes, generating images, and applying edge detection to create corresponding "fake" sketches.

Key Results

SketchInverter significantly outperformed baseline models like Pix2pix and EdgeGAN across multiple metrics. On the Sketchy Database, SketchInverter achieved a Fréchet Inception Distance (FID) of 56.71, nearly doubling the performance of the best baseline (Pix2pix-Sep at 107.59). Lower FID scores indicate higher image quality and diversity.

Qualitative results showing SketchInverter's ability to generate photo-realistic images from abstract sketches across various classes.

In classification tests, images generated by SketchInverter were recognized correctly by a pre-trained classifier 98.8% of the time, compared to just 17.4% for the multi-class Pix2pix-Mix. Qualitatively, the generated images show sharper textures, more realistic lighting, and better alignment with the original sketch's pose and orientation compared to previous state-of-the-art methods.

Discussion and Conclusion

The core insight of SketchInverter is that the latent space of a pretrained GAN acts as a manifold of "natural images." By forcing the output to live on this manifold, the model ensures the production of realistic images. By utilizing a pretrained BigGAN, the authors effectively "borrow" the knowledge learned from millions of ImageNet photos without needing a paired sketch for every single one.

While highly effective, the model is currently limited by the capabilities of the underlying pretrained generator and requires a user-provided class label. Future work may involve automatically predicting the class from the sketch and utilizing more modern architectures like StyleGAN-XL to achieve higher resolutions.

Summary updated

Abstract

This paper proposes the first GAN inversion-based method for multi-class sketch-based image generation (MCSBIG). MC-SBIG is a challenging task that requires strong prior knowledge due to the significant domain gap between sketches and natural images. Existing learning-based approaches rely on a large-scale paired dataset to learn the mapping between these two image modalities. However, since the public paired sketch-photo data are scarce, it is struggling for learning-based methods to achieve satisfactory results. In this work, we introduce a new approach based on GAN inversion, which can utilize a powerful pretrained generator to facilitate image generation from a given sketch. Our GAN inversion-based method has two advantages: 1. it can freely take advantage of the prior knowledge of a pretrained image generator; 2. it allows the proposed model to focus on learning the mapping from a sketch to a low-dimension latent code, which is a much easier task than directly mapping to a high-dimension natural image. We also present a novel shape loss to improve generation quality further. Extensive experiments are conducted to show that our method can produce sketch-faithful and photo-realistic images and significantly outperform the baseline methods.
BibTeX
@inproceedings{an2023sketchinverter,
  author = {Zirui An and Jingbo Yu and Runtao Liu and Chuang Wang and Qian Yu},
  title = {{SketchInverter: Multi-Class Sketch-Based Image Generation via GAN Inversion}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=EQ8TGZzm9F},
}
Figure