Synthesizing human-like sketches from natural images using a conditional convolutional decoder

Moritz Kampelmuhler, Axel PinzWACV 2020

TL;DR

This paper introduces a novel convolutional neural network architecture capable of generating human-like sketches from natural images, even those with cluttered backgrounds. By employing a perceptual similarity loss and conditioning the decoder on object labels, the model learns to capture abstract shape representations, bridging the gap between detailed images and simplified sketches.

What Problem Does This Paper Address?

Humans can effortlessly communicate complex ideas through sketches, which are highly abstract yet convey essential shape information. While edge detection can extract basic outlines, it fails to capture the "human-like" quality of sketches, which often emphasize salient features and conceptual representations over precise detail. Existing methods for image-to-sketch synthesis often struggle with this abstraction, particularly when dealing with complex, real-world images. This paper aims to bridge this gap by developing a deep learning model that can synthesize human-like sketches from natural images, demonstrating that Convolutional Neural Networks (CNNs) can learn and utilize abstract shape representations.

Methods

The core of the proposed method is a fully convolutional encoder-decoder architecture. The encoder, based on a pre-trained VGG16 network, transforms the input image into a compact feature representation (bottleneck). The decoder then reconstructs a sketch from this bottleneck. Key innovations include:

  • Perceptual Similarity Loss: Instead of traditional pixel-wise losses (like MSE), which are sensitive to small shifts and can produce blurry results, the authors use a loss function based on the cosine similarity of feature activations from an independently trained AlexNet classifier. This metric is more robust to minor variations and better captures perceptual similarity.
  • Conditional Decoding with AdaIN: The decoder is conditioned on the object's class label using Adaptive Instance Normalization (AdaIN). This allows the model to inject "shape priors" at various abstraction levels within the decoder, guiding it to generate sketches appropriate for the specific object class. This is achieved by learning class-specific statistics for normalization.
  • Skip Connections: To incorporate finer details and improve reconstruction quality, skip connections are implemented, concatenating encoder activations with corresponding decoder layers.

Datasets

The primary dataset used is the Sketchy Database [26]. This dataset comprises 125 object categories, with natural images sourced from ImageNet and multiple human-generated sketches for each image. The authors note that sketchers were given only a brief time to view the images, encouraging abstraction. For training and evaluation, they used a 90/10 train/test split, splitting on images rather than sketch-image pairs to ensure unseen images in the test set.

Key Insights and Ideas

  • CNNs Learn Implicit Shape: The research strongly suggests that CNNs trained for image classification implicitly learn valuable shape representations, which can be leveraged for tasks like sketch synthesis.
  • Perceptual Loss for Domain Gap: The perceptual similarity metric is crucial for bridging the significant domain gap between natural images and abstract sketches, outperforming pixel-wise losses.
  • Conditional Decoding Enhances Abstraction: Conditioning the decoder on object labels with AdaIN significantly improves the quality and class-appropriateness of the generated sketches.
  • Human-like Abstraction is Key: The goal is not to produce photorealistic edge maps but to mimic the human ability to abstract and convey the essence of an object's shape.

Key Results

The proposed method achieves a Top-1 accuracy of 66.6% and Top-5 accuracy of 85.6% on a ResNet-34 classifier trained on generated sketches, significantly outperforming baselines like MSE loss and edge detection methods. Qualitative results show that the generated sketches are visually appealing and capture the essential shapes of objects. A user study further validated the visual quality, with participants preferring the generated sketches from the full model over those with fewer components.

Limitations and Future Work

The authors acknowledge that the model's performance can be affected by unusual object poses, occlusions, and poor visibility, which are inherent challenges in sketch generation. They also note that for classes with low intra-class variance in the dataset (e.g., sailboats), the results might appear to indicate mode collapse, though this is attributed to the data itself. Future work could focus on increasing the perceptual quality of generated samples, improving robustness to challenging poses and images, and potentially exploring more sophisticated conditioning mechanisms.

Summary updated

BibTeX
@inproceedings{kampelmuhler2020synthesizing,
  author = {Moritz Kampelmuhler and Axel Pinz},
  title = {{Synthesizing human-like sketches from natural images using a conditional convolutional decoder}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure