Abstract
Humans can envision a realistic photo given a free-hand sketch that is not only spatially imprecise and geometrically distorted but also without colors and visual details. We study unsupervised sketch-to-photo synthesis for the first time, learning from unpaired sketch-photo data where the target photo for a sketch is unknown during training. Existing works only deal with style change or spatial deformation alone, synthesizing photos from edge-aligned line drawings or transforming shapes within the same modality, e.g., color images. Our key insight is to decompose unsupervised sketch-to-photo synthesis into a two-stage translation task: First shape translation from sketches to grayscale photos and then content enrichment from grayscale to color photos. We also incorporate a self-supervised denoising objective and an attention module to handle abstraction and style variations that are inherent and specific to sketches. Our synthesis is sketch-faithful and photo-realistic to enable sketch-based image retrieval in practice. An exciting corollary product is a universal and promising sketch generator that captures human visual perception beyond the edge map of a photo.
Introduction: The Gap Between Sketch and Photo
When a human views a quick, hand-drawn sketch, they can intuitively reconstruct the intended 3D object despite wobbly lines, missing textures, or incorrect proportions. For computer vision systems, however, this task is significantly more challenging. Most existing sketch-to-photo systems rely on paired data or perfectly aligned edge maps. In real-world scenarios, sketches are typically "unpaired" and "misaligned," characterized by significant spatial distortion and a lack of color or texture.
Standard unsupervised image-to-image translation tools, such as CycleGAN, often fail in this domain. They struggle to map pixels across the massive domain gap where both geometry is distorted and style is absent, frequently resulting in unrealistic "hallucinated" artifacts.
Methodology: Decomposing the Challenge
The researchers propose a novel two-stage framework to bridge the gap between abstract sketches and realistic photos.
Stage 1: Shape Translation
The first stage focuses on rectifying geometric distortions by translating the sketch into a realistic grayscale photo. To handle the variety of human drawing styles, the model employs Noise Sketch Composition. By adding artificial noise—such as messy strokes or random patches—and requiring the model to reconstruct a clean sketch through a self-supervised denoising objective, the system becomes robust to abstraction. Additionally, an Inverse Attention module identifies and suppresses dense, distracting stroke areas, allowing the model to focus on the essential underlying structure.
Stage 2: Content Enrichment
Once the shape is established in grayscale, the second stage "hallucinates" color and texture. This stage utilizes an encoder-decoder architecture with Adaptive Instance Normalization (AdaIN). This allows for flexible synthesis, where the model can either generate a default colorization or adopt the style of an optional reference photo (e.g., applying a specific material or color to a synthesized shoe).
Key Results
The proposed model demonstrates superior performance both quantitatively and qualitatively. On the ShoeV2 dataset, it achieved a Fréchet Inception Distance (FID) score of 48.73, significantly outperforming unsupervised baselines like CycleGAN (79.35) and even supervised methods like Pix2Pix (65.09).
Qualitatively, user studies showed a strong preference for the authors' results. The model proved robust to partial sketches and even cross-category inputs. Furthermore, the system effectively bridged the domain gap for Sketch-Based Image Retrieval (SBIR), achieving 37.2% top-5 accuracy without explicit supervision.
Conclusion
This work represents the first successful unsupervised framework for sketch-to-photo synthesis. By decomposing the problem into shape translation and content enrichment, the model overcomes the limitations of previous methods that struggle with geometric distortion. While currently focused on specific object categories, this approach paves the way for more advanced human-computer interaction tools and universal sketch generators that better mimic human artistic intent.
Summary updated