Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation

Arnab Ghosh, Richard Zhang, Puneet K. Dokania, Oliver Wang, Alexei A. Efros, Philip H. S. Torr, Eli Shechtman

Abstract: We propose an interactive GAN-based sketch-to-image translation method that helps novice users easily create images of simple objects. The user starts with a sparse sketch and a desired object category, and the network then recommends its plausible completion(s) and shows a corresponding synthesized image. This enables a feedback loop, where the user can edit the sketch based on the network's recommendations, while the network is able to better synthesize the image that the user might have in mind. In order to use a single model for a wide array of object classes, we introduce a gating-based approach for class conditioning, which allows us to generate distinct classes without feature mixing, from a single generator network.

Introduction

Most existing sketch-to-image tools operate under the assumption that the user can provide a high-quality, complete edge map. For the average person, however, drawing accurate proportions and 3D perspectives from scratch is a significant barrier. Furthermore, standard image-to-image translation models (such as pix2pix) often struggle with multi-class data. When trained to generate multiple distinct categories—such as oranges and basketballs—these models frequently produce "mushy" results that fail to capture the specific textures and details unique to each class.

Interactive Sketch & Fill: The system completes partial sketches and translates them into various object classes based on user selection.

To address these issues, we introduce "Interactive Sketch & Fill," a framework designed to act as an AI drawing assistant. It tackles two primary challenges: the "blank canvas" problem by providing real-time shape recommendations, and the feature mixing problem by utilizing a novel gating mechanism that prevents object categories from "bleeding" into one another.

Methods

Our approach decouples the translation task into a two-stage pipeline, separating the generation of geometry from the synthesis of appearance.

The Two-Stage Pipeline

  1. Shape Completion ($G_S$): This stage takes partial user strokes and predicts a completed outline. It functions as a recommender system, providing the user with a gray "shadow" of potential final shapes. This allows users to refine their drawings based on the AI's suggestions.
  2. Appearance Synthesis ($G_A$): This stage takes the completed outline and a class label to generate the final rendered image.
The two-stage pipeline: $G_S$ generates a complete outline from sparse strokes, and $G_A$ synthesizes the final image using class-conditioned gating.

Soft Gating Mechanisms

To solve the multi-class problem within a single model, we introduce Soft Gating. Rather than naively concatenating a class label onto the input, we utilize a small external gating network. This network predicts a set of parameters ($\alpha$) that act as a "dimmer switch" for every channel in the main generator's residual blocks.

For instance, if the user selects "pineapple," the gating network activates channels responsible for prickly textures while deactivating those for smooth surfaces. We found that channel-wise gating—modulating each channel individually—performed significantly better than block-wise gating.

Datasets and Evaluation

We introduced a new Multi-class Outline Dataset containing 2,000 images across 10 classes (Basketball, Soccer, Watermelon, Orange, Cookie, Moon, Strawberry, Pineapple, Cupcake, and Fried Chicken). These classes were chosen because they often share similar circular base shapes, providing a rigorous "stress test" for class conditioning. We also evaluated the model on standard benchmarks like edges2shoes and CelebA-HQ.

Key Results

  • Quantitative Performance: In multi-class tests, our gated model achieved a classification accuracy of 97.38% for generated images, compared to approximately 73% for standard one-stage approaches.
  • Realism: Using a "Visual Turing Test" via Amazon Mechanical Turk, the ChannelGate model fooled human observers significantly more often than naive concatenation methods.
  • Creative Generalization: The model demonstrates an ability to apply textures to unusual shapes. For example, if a user draws a heart-shaped outline but selects the "pineapple" class, the gating mechanism successfully applies pineapple textures to that heart shape.

Conclusion

By decoupling shape and texture and employing a channel-wise gating mechanism, we have developed a robust tool for novice-friendly image synthesis. While the current model focuses on single objects against plain backgrounds, it proves that depth and smart conditioning are more vital than raw network width. Future work will explore extending this gating approach to more complex domains, such as full-body human synthesis or architectural design, where strict separation of styles is required within a single model.

Summary updated

Abstract

We propose an interactive GAN-based sketch-to-image translation method that helps novice users easily create images of simple objects. The user starts with a sparse sketch and a desired object category, and the network then recommends its plausible completion(s) and shows a corresponding synthesized image. This enables a feedback loop, where the user can edit the sketch based on the network's recommendations, while the network is able to better synthesize the image that the user might have in mind. In order to use a single model for a wide array of object classes, we introduce a gating-based approach for class conditioning, which allows us to generate distinct classes without feature mixing, from a single generator network.
BibTeX
@inproceedings{ghosh2019interactive,
  author = {Arnab Ghosh and Richard Zhang and Puneet K. Dokania and Oliver Wang and Alexei A. Efros and Philip H. S. Torr and Eli Shechtman},
  title = {{Interactive Sketch \& Fill: Multiclass Sketch-to-Image Translation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=INQHaiv5KIo},
}
Figure