Unsupervised Sketch to Photo Synthesis

Runtao Liu, Qian Yu, Stella X. Yu

Abstract

Humans can envision a realistic photo given a free-hand sketch that is not only spatially imprecise and geometrically distorted but also without colors and visual details. We study unsupervised sketch-to-photo synthesis for the first time, learning from unpaired sketch-photo data where the target photo for a sketch is unknown during training. Existing works only deal with style change or spatial deformation alone, synthesizing photos from edge-aligned line drawings or transforming shapes within the same modality, e.g., color images. Our key insight is to decompose unsupervised sketch-to-photo synthesis into a two-stage translation task: First shape translation from sketches to grayscale photos and then content enrichment from grayscale to color photos. We also incorporate a self-supervised denoising objective and an attention module to handle abstraction and style variations that are inherent and specific to sketches. Our synthesis is sketch-faithful and photo-realistic to enable sketch-based image retrieval in practice. An exciting corollary product is a universal and promising sketch generator that captures human visual perception beyond the edge map of a photo.

Introduction: The Gap Between Sketch and Photo

When a human views a quick, hand-drawn sketch, they can intuitively reconstruct the intended 3D object despite wobbly lines, missing textures, or incorrect proportions. For computer vision systems, however, this task is significantly more challenging. Most existing sketch-to-photo systems rely on paired data or perfectly aligned edge maps. In real-world scenarios, sketches are typically "unpaired" and "misaligned," characterized by significant spatial distortion and a lack of color or texture.

The spectrum of abstractness from RGB photos to human sketches, highlighting the gap between computer vision and human perception.

Standard unsupervised image-to-image translation tools, such as CycleGAN, often fail in this domain. They struggle to map pixels across the massive domain gap where both geometry is distorted and style is absent, frequently resulting in unrealistic "hallucinated" artifacts.

Methodology: Decomposing the Challenge

The researchers propose a novel two-stage framework to bridge the gap between abstract sketches and realistic photos.

Stage 1: Shape Translation

The first stage focuses on rectifying geometric distortions by translating the sketch into a realistic grayscale photo. To handle the variety of human drawing styles, the model employs Noise Sketch Composition. By adding artificial noise—such as messy strokes or random patches—and requiring the model to reconstruct a clean sketch through a self-supervised denoising objective, the system becomes robust to abstraction. Additionally, an Inverse Attention module identifies and suppresses dense, distracting stroke areas, allowing the model to focus on the essential underlying structure.

Stage 2: Content Enrichment

Once the shape is established in grayscale, the second stage "hallucinates" color and texture. This stage utilizes an encoder-decoder architecture with Adaptive Instance Normalization (AdaIN). This allows for flexible synthesis, where the model can either generate a default colorization or adopt the style of an optional reference photo (e.g., applying a specific material or color to a synthesized shoe).

Overview of the two-stage synthesis pipeline and qualitative comparison with existing unsupervised and supervised methods across different data alignment scenarios.

Key Results

The proposed model demonstrates superior performance both quantitatively and qualitatively. On the ShoeV2 dataset, it achieved a Fréchet Inception Distance (FID) score of 48.73, significantly outperforming unsupervised baselines like CycleGAN (79.35) and even supervised methods like Pix2Pix (65.09).

Qualitatively, user studies showed a strong preference for the authors' results. The model proved robust to partial sketches and even cross-category inputs. Furthermore, the system effectively bridged the domain gap for Sketch-Based Image Retrieval (SBIR), achieving 37.2% top-5 accuracy without explicit supervision.

Conclusion

This work represents the first successful unsupervised framework for sketch-to-photo synthesis. By decomposing the problem into shape translation and content enrichment, the model overcomes the limitations of previous methods that struggle with geometric distortion. While currently focused on specific object categories, this approach paves the way for more advanced human-computer interaction tools and universal sketch generators that better mimic human artistic intent.

Summary updated

Abstract

Humans can envision a realistic photo given a free-hand sketch that is not only spatially imprecise and geometrically distorted but also without colors and visual details. We study unsupervised sketch to photo synthesis for the first time, learning from unpaired sketch and photo data where the target photo for a sketch is unknown during train- ing. Existing works only deal with either style difference or spatial deformation alone, synthesizing photos from edge-aligned line drawings or transforming shapes within the same modality, e.g., color images. Our insight is to decompose the unsupervised sketch to photo syn- thesis task into two stages of translation: First shape translation from sketches to grayscale photos and then content enrichment from grayscale to color photos. We also incorporate a self-supervised denoising objec- tive and an attention module to handle abstraction and style variations that are specific to sketches. Our synthesis is sketch-faithful and photo- realistic, enabling sketch-based image retrieval and automatic sketch gen- eration that captures human visual perception beyond the edge map of a photo.
BibTeX
@inproceedings{liu2020unsupervised,
  author = {Runtao Liu and Qian Yu and Stella X. Yu},
  title = {{Unsupervised Sketch to Photo Synthesis}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=VDzAYQCbhjd},
}
Figure