SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks

Alex Lamb, Sherjil Ozair, Vikas Verma, David Ha

TL;DR

Deep neural networks are notoriously reliant on local textures and high-frequency details, often failing to recognize objects when those details are stripped away. This paper introduces SketchTransfer, a new benchmark that challenges models to classify human sketches (from QuickDraw) using only labeled real images (from CIFAR-10) and unlabeled sketches. The results show a massive "abstraction gap": while humans generalize to doodles effortlessly, state-of-the-art domain adaptation techniques still fall significantly short of supervised performance.

What Problem Does This Paper Address?

Humans possess a remarkable ability called detail-invariance. We can recognize a "dog" whether it’s a high-resolution photograph, a cartoon, or a 20-second "QuickDraw" doodle. We do this by learning high-level abstractions—like the specific shape of a cat's ears—while discarding irrelevant textures.

In contrast, deep networks are "texture-biased." Research shows they often classify objects based on local patterns (like fur or skin texture) rather than global structure. When these details are removed, model performance collapses. The authors argue that current machine learning lacks a robust way to measure and encourage this human-like abstraction. They propose the SketchTransfer task to quantify how well models can transfer knowledge from a detail-rich domain (real photos) to a detail-sparse domain (sketches).

Methods

The authors evaluate several state-of-the-art Unsupervised Domain Adaptation (UDA) and self-supervised learning techniques to see if they can bridge the gap between photos and sketches:

  • Adversarial Domain Adaptation: Using a discriminator to make the internal representations of real images and sketches look identical to the model.
  • Virtual Mixup Training (VMT): A consistency-based approach that interpolates between images and their "pseudo-labels" to smooth out decision boundaries.
  • Rotation Prediction: A self-supervised task where the model predicts if an image has been rotated (0, 90, 180, or 270 degrees), forcing it to learn more structural features.
  • CyCada / CycleGAN: Using generative models to "translate" real images into the style of sketches (and vice versa) to provide more familiar training data.
  • Dirt-T: An iterative refinement procedure that enforces the "cluster assumption"—the idea that decision boundaries should not pass through high-density regions of the target data.

Datasets

The paper introduces the SketchTransfer dataset, which creates a correspondence between two existing popular datasets:

  1. Source Domain (Real Images): CIFAR-10, consisting of 60,000 32x32 color images across 10 classes (airplane, automobile, bird, etc.).
  2. Target Domain (Sketches): QuickDraw, a collection of millions of sketches drawn by humans in under 20 seconds.

The authors mapped specific QuickDraw categories to CIFAR-10 classes (e.g., mapping "Police Car" and "Car" to "Automobile"). Because QuickDraw lacks a "Deer" category, the final SketchTransfer test set contains 9 classes.

Key Insights and Ideas

  • The "Texture Crutch": Standard models trained on CIFAR-10 achieve only ~23% accuracy on sketches. This confirms that these models rely on color and texture features that simply don't exist in a line drawing.
  • Unlabeled Data is Essential but Insufficient: Using unlabeled sketches during training (via UDA) improves performance significantly (up to ~59%), but it still leaves a 28% gap compared to models trained directly on labeled sketches.
  • Semantic vs. Pixel Similarity: Nearest-neighbor analysis shows that while raw pixel-space comparisons are useless for sketches, models trained with domain adaptation start to group sketches by semantic meaning (e.g., grouping different types of birds together), even if they can't quite classify them perfectly yet.
  • The Difficulty of "Salient" Features: For humans, the difference between a dog and a cat sketch might just be the ear shape. Models struggle to identify these "highly salient" but visually small features without explicit supervision.

Key Results

The paper provides a clear hierarchy of performance on the SketchTransfer task:

  • Random Guessing: 11% accuracy.
  • Source Only (CIFAR-10 only): ~23.3% accuracy.
  • **Best UDA (Virtual Mixup + Domain Adv.
  • DIRT-T): ~58.9% accuracy.**
  • Supervised (Trained on labeled sketches): ~87.3% accuracy.

While the best UDA methods are much better than random, the fact that they lag nearly 30% behind supervised models highlights that current "state-of-the-art" transfer learning still struggles with extreme changes in level-of-detail.

Limitations and Future Work

  • Dataset Complexity: CIFAR-10 images are small (32x32) and usually centered. The authors suggest that moving to more complex datasets with varied backgrounds and contexts would make the task even more challenging and realistic.
  • The "Middle Ground": Sketches are an extreme case of detail removal. Future work could explore "intermediate" domains, such as cartoons or shaded illustrations, to see if they can serve as a stepping stone for models to learn better abstractions.
  • Alternative Approaches: The authors suggest exploring Artist Agents (models that learn to "draw" an image stroke-by-stroke) or Meta-Learning as potential ways to force models to understand the fundamental structure of objects rather than just their surface textures.

Summary updated

Abstract

Deep networks have achieved excellent results in perceptual tasks, yet their ability to generalize to variations not seen during training has come under increasing scrutiny. In this work we focus on their ability to have invariance towards the presence or absence of details. For example, humans are able to watch cartoons, which are missing many visual details, without being explicitly trained to do so. As another example, 3D rendering software is a relatively recent development, yet people are able to understand such rendered scenes even though they are missing details (consider a film like Toy Story). The failure of ma- chine learning algorithms to do this indicates a significant gap in generalization between human abilities and the abilities of deep networks. We propose a dataset that will make it easier to study the detail-invariance problem concretely. We produce a concrete task for this: SketchTransfer, and we show that state-of-the-art domain transfer algorithms still struggle with this task. The state-of-the-art technique which achieves over 95% on MNIST → SVHN transfer only achieves 59% accuracy on the SketchTransfer task, which is much better than random (11% accuracy) but falls short of the 87% accuracy of a classifier trained directly on labeled sketches. This indicates that this task is approachable with today’s best methods but has substantial room for improvement.
BibTeX
@inproceedings{lamb2020sketchtransfer,
  author = {Alex Lamb and Sherjil Ozair and Vikas Verma and David Ha},
  title = {{SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=8PPpBZcvOdj},
}

Loading related content...

Figure