TL;DR
This paper introduces an interactive system for sketch-to-image translation that allows novice users to create object images by drawing sparse sketches. The system uses a two-stage approach: first, it intelligently completes the user's sketch, and then it generates a realistic image from the completed outline. A novel "gating" mechanism is proposed to enable a single model to generate diverse object classes effectively, overcoming limitations of previous methods.
What Problem Does This Paper Address?
Creating realistic images from scratch, especially for users without strong artistic skills, is challenging. Existing GAN-based image translation methods often require a complete, abstract input (like a dense edge map or segmentation map), which is difficult for untrained users to provide. Furthermore, these methods are typically limited to generating images of a single object class, requiring separate models for each. This paper aims to bridge this gap by developing an interactive system that assists users by completing their partial sketches and generating images for multiple object classes using a single, unified model.
Methods
The proposed system employs a two-stage approach:
- Shape Completion: A shape generator ($G_S$) takes a sparse user sketch as input and predicts plausible completions of the object's outline. This stage is designed to be interactive, providing feedback to the user as they draw. To train this generator, the authors simulate partial sketches by removing random patches from complete outlines. They also introduce "simplified edges" as an intermediate representation, which are closer to human sketches.
- Appearance Synthesis: An appearance generator ($G_A$) takes the completed outline from $G_S$ and generates the final image. The key novelty here is the gating mechanism for class conditioning. Instead of simply concatenating class information to the input or feature maps (which can lead to feature mixing and confusion between classes), the authors propose a soft gating mechanism. This mechanism uses a small, external gating network conditioned on the class label to predict parameters that modulate the features within the main generator and discriminator networks. This allows the network to selectively activate or deactivate specific blocks or channels based on the desired class, leading to cleaner separation and better generation. They explore both block-wise and channel-wise gating, finding channel-wise gating to be most effective.
Datasets
- New Dataset: The authors introduce a new dataset consisting of rough outlines for 10 simple object classes (basketball, chicken, cookie, cupcake, moon, orange, soccer, strawberry, watermelon, pineapple) with white backgrounds. These outlines were generated by finding the largest blob in images and smoothing its boundary.
- Existing Datasets:
- UTZappos Shoes dataset (for single-class generation).
- CelebA-HQ dataset (for single-class generation).
Key Insights and Ideas
- Two-Stage Approach is Crucial: Separating shape completion from appearance synthesis significantly improves results. The intermediate shape completion provides a more robust geometric representation, making the subsequent image generation task easier and more accurate.
- Gating for Effective Class Conditioning: Simple concatenation of class labels is insufficient for multi-class generation with a single generator. The proposed gating mechanism effectively injects class information by selectively controlling feature activations, preventing class confusion and leading to higher accuracy and realism.
- Channel-wise Gating is Superior: Channel-wise soft gating, which allows for finer control over feature activations, yields the best results in terms of both classification accuracy and visual realism compared to block-wise gating or other conditioning methods.
- Interactive Feedback Loop: The system's interactive nature, where the user can edit sketches based on the model's suggestions, is key to its usability and effectiveness in guiding the generation process.
Key Results
- Improved Accuracy: The proposed two-stage approach with channel-wise gating achieves significantly higher classification accuracy on generated images compared to one-stage methods and other conditioning techniques (e.g., 97.38% for multi-class outline-to-image with the proposed method vs. 88.74% for a one-stage approach).
- Higher Realism: The gating mechanism leads to more realistic generations, as evidenced by "Visual Turing tests" where human evaluators were less able to distinguish generated images from real ones (e.g., AMT fooling rate of 23.4% for channel-wise gating vs. 17.7% for per-class generators).
- Generalization: The model demonstrates the ability to generate plausible images even when the input circle outline doesn't strictly correspond to the object's shape (e.g., generating a strawberry from a circle).
- Architecture Efficiency: The "SkinnyResNet" architecture, a more efficient variant of the generator, also benefits from the gating mechanism and outperforms the more complex Encoder-Decoder architecture in some metrics.
Limitations and Future Work
- Limitations:
- The paper focuses on simple objects. Extending the system to more complex scenes or objects with intricate details might be challenging.
- While the system is interactive, the quality of the generated image still depends on the user's ability to provide a reasonable initial sketch.
- Future Work:
- The authors suggest exploring more sophisticated gating mechanisms or architectural modifications.
- Extending the system to handle more complex scenes and object categories is a natural next step.
- Further improvements in the realism and diversity of generated images are always desirable.
Summary updated