An Unpaired Sketch-to-Photo Translation Model

Runtao Liu, Qian Yu, Stella X. Yu

Abstract Humans can envision a realistic photo given a free-hand sketch that is not only spatially imprecise and geometrically distorted but also without colors and visual details. We study unsupervised sketch-to-photo synthesis for the first time, learning from unpaired sketch-photo data where the target photo for a sketch is unknown during training. Existing works only deal with style change or spatial deformation alone, synthesizing photos from edge-aligned line drawings or transforming shapes within the same modality, e.g., color images. Our key insight is to decompose unsupervised sketch-to-photo synthesis into a two-stage translation task: First shape translation from sketches to grayscale photos and then content enrichment from grayscale to color photos. We also incorporate a self-supervised denoising objective and an attention module to handle abstraction and style variations that are inherent and specific to sketches. Our synthesis is sketch-faithful and photo-realistic to enable sketch-based image retrieval in practice. An exciting corollary product is a universal and promising sketch generator that captures human visual perception beyond the edge map of a photo.

Introduction

Translating a free-hand sketch into a realistic photo is a long-standing challenge in computer vision. While existing solutions often rely on paired data—where each sketch is perfectly matched to a photo—or clean edge maps, real-world human sketches present unique difficulties. They are characterized by significant geometric distortion, high levels of abstraction, and "noisy" strokes that do not correspond to physical features. Furthermore, collecting large-scale paired datasets is prohibitively expensive. This research introduces the first unsupervised model capable of transforming these messy, unaligned sketches into high-fidelity color photos by learning from independent sets of sketches and images.

Proposed Method

The core innovation lies in a two-stage translation pipeline that decouples shape rectification from content enrichment. This separation simplifies the learning process for unsupervised Generative Adversarial Networks (GANs), which often struggle to handle simultaneous geometric and stylistic transformations.

Overview of the two-stage unpaired sketch-to-photo translation pipeline compared against existing baselines.
  1. Stage 1: Shape Translation (Sketch $\to$ Grayscale Photo): This stage focuses on correcting the geometric distortions of the input sketch. To handle the inherent noise in human drawings, the authors employ a self-supervised denoising objective, where the model learns to reconstruct clean sketches from intentionally corrupted versions. Additionally, a specialized de-noising attention module identifies and suppresses activations in overly dense or distracting stroke areas, allowing the model to focus on the essential structure.
  2. Stage 2: Content Enrichment (Grayscale $\to$ Color Photo): Once a geometrically sound grayscale image is produced, this stage adds texture, shading, and color. Using an encoder-decoder architecture with Adaptive Instance Normalization (AdaIN), the model can optionally use a reference photo to guide the final visual style, such as the material or color of a shoe.

Experimental Results

The model was rigorously evaluated on the ShoeV2 and ChairV2 datasets, which feature fine-grained details and significant abstraction.

Quantitative comparison on ShoeV2 and ChairV2 datasets showing superior FID and Quality scores for the proposed model.

As shown in the table above, the proposed model significantly outperforms standard unsupervised translation baselines like CycleGAN and MUNIT. On the ShoeV2 dataset, it achieved a Fréchet Inception Distance (FID) of 48.73, a substantial improvement over CycleGAN’s 79.35. Qualitative user studies further confirmed these results, with participants preferring the proposed model's outputs approximately 50% of the time when compared against multiple baselines that often produced unrecognizable or distorted shapes.

Beyond synthesis, the model's effectiveness was demonstrated in Sketch-Based Image Retrieval (SBIR). By first translating a query sketch into a photo, the system achieved a top-5 retrieval accuracy of 37.2% without any explicit retrieval-specific training.

Discussion and Insights

A key finding of this work is that grayscale images serve as a superior intermediate representation compared to edge maps. Grayscale provides richer learning signals, such as shading and gradients, which are crucial for understanding 3D structure. Furthermore, the model's bidirectional nature allows it to function as a "Universal Sketcher," generating sketches from photos that mimic human perceptual contours rather than simple mathematical edges.

Conclusion and Limitations

While the two-stage decomposition effectively handles unpaired sketch-to-photo translation, the model currently performs best when trained on specific categories (e.g., shoes or chairs). Future work will focus on expanding the model's semantic understanding to handle complex, multi-object scenes and improving its ability to generalize across diverse object categories without category-specific training.

Summary updated

Abstract

Humans can envision a realistic photo given a free-hand sketch that is not only spatially imprecise and geometrically distorted but also without colors and visual details. We study unsupervised sketch-to-photo synthesis for the first time, learning from unpaired sketch-photo data where the target photo for a sketch is unknown during training. Existing works only deal with style change or spatial deformation alone, synthesizing photos from edge-aligned line drawings or transforming shapes within the same modality, e.g., color images. Our key insight is to decompose unsupervised sketch-to-photo synthesis into a two-stage translation task: First shape translation from sketches to grayscale photos and then content enrichment from grayscale to color photos. We also incorporate a self-supervised denoising objective and an attention module to handle abstraction and style variations that are inherent and specific to sketches. Our synthesis is sketch-faithful and photo-realistic to enable sketch-based image retrieval in practice. An exciting corollary product is a universal and promising sketch generator that captures human visual perception beyond the edge map of a photo.
BibTeX
@inproceedings{liu2020unpaired,
  author = {Runtao Liu and Qian Yu and Stella X. Yu},
  title = {{An Unpaired Sketch-to-Photo Translation Model}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=VDzAYQCbhjd},
}
Figure