SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks

Alex Lamb, Sherjil Ozair, Vikas Verma, David Ha

Abstract

Deep networks have achieved excellent results in perceptual tasks, yet their ability to generalize to variations not seen during training has come under increasing scrutiny. In this work, we focus on their ability to have invariance towards the presence or absence of details. For example, humans are able to watch cartoons, which are missing many visual details, without being explicitly trained to do so. The failure of machine learning algorithms to do this indicates a significant gap in generalization between human abilities and the abilities of deep networks. We propose a dataset, SketchTransfer, to study the detail-invariance problem concretely. We show that state-of-the-art domain transfer algorithms still struggle with this task, achieving 59% accuracy compared to the 87% accuracy of a classifier trained directly on labeled sketches. This indicates that this task is approachable with today's best methods but has substantial room for improvement.

Introduction: The Detail-Invariance Problem

Humans possess a remarkable ability called detail-invariance. We can recognize a "dog" whether it is a high-resolution photograph, a cartoon, or a 20-second "QuickDraw" doodle. We achieve this by learning high-level abstractions—like the specific shape of a cat's ears—while discarding irrelevant textures.

Figure 1: Comparison of a low-detail sketch versus a high-detail sketch of a dollar bill, illustrating the concept of detail-invariance.

In contrast, deep networks are often "texture-biased," relying on local patterns (like fur or skin texture) rather than global structure. When these details are removed, model performance collapses. The SketchTransfer task quantifies how well models can transfer knowledge from a detail-rich domain (real photos) to a detail-sparse domain (sketches).

The SketchTransfer Dataset

The SketchTransfer dataset creates a correspondence between two existing benchmarks to test cross-domain abstraction:

  1. Source Domain (Real Images): CIFAR-10, consisting of 60,000 32x32 color images across 10 classes.
  2. Target Domain (Sketches): QuickDraw, a collection of millions of human-drawn sketches.

The authors mapped specific QuickDraw categories to CIFAR-10 classes (e.g., mapping "Police Car" to "Automobile"). Because QuickDraw lacks a "Deer" category, the final SketchTransfer test set contains 9 classes.

Figure 2: Examples from the SketchTransfer dataset showing the mapping between real images (top rows) and human sketches (bottom rows) for Plane, Cat, and Dog classes.

Methods and Evaluation

The study evaluates several state-of-the-art Unsupervised Domain Adaptation (UDA) and self-supervised learning techniques to bridge the "abstraction gap":

  • Adversarial Domain Adaptation: Using a discriminator to align internal representations of real images and sketches.
  • Virtual Mixup Training (VMT): A consistency-based approach that interpolates between images and pseudo-labels to smooth decision boundaries.
  • Rotation Prediction: A self-supervised task forcing the model to learn structural features by predicting image orientation.
  • CyCada / CycleGAN: Generative models that translate real images into the style of sketches.
  • Dirt-T: An iterative refinement procedure that enforces the "cluster assumption" on target data.

Key Results

The results highlight a significant performance gap between current UDA methods and supervised learning:

Method Accuracy
Random Guessing 11%
Source Only (CIFAR-10 only) ~23.3%
**Best UDA (VMT + Adv.
  • DIRT-T) | ~58.9%** | | Supervised (Trained on labeled sketches) | ~87.3% |

Standard models trained only on CIFAR-10 fail significantly (~23% accuracy), confirming a heavy reliance on color and texture. While UDA methods improve performance to nearly 59%, they still lag 28% behind supervised models, indicating that current transfer learning struggles with extreme changes in the level of detail.

Insights and Future Work

Nearest-neighbor analysis shows that while raw pixel-space comparisons are useless for sketches, models trained with domain adaptation begin to group sketches by semantic meaning. However, models still struggle to identify "highly salient" but visually small features (like ear shapes) that humans use to distinguish classes.

Future work may explore "intermediate" domains, such as cartoons, to serve as a stepping stone for abstraction. Additionally, the authors suggest exploring Artist Agents (models that learn to "draw" stroke-by-stroke) or Meta-Learning to force models to understand fundamental structure rather than surface textures.

Summary updated

Abstract

Deep networks have achieved excellent results in perceptual tasks, yet their ability to generalize to variations not seen during training has come under increasing scrutiny. In this work we focus on their ability to have invariance towards the presence or absence of details. For example, humans are able to watch cartoons, which are missing many visual details, without being explicitly trained to do so. As another example, 3D rendering software is a relatively recent development, yet people are able to understand such rendered scenes even though they are missing details (consider a film like Toy Story). The failure of machine learning algorithms to do this indicates a significant gap in generalization between human abilities and the abilities of deep networks. We propose a dataset that will make it easier to study the detail-invariance problem concretely. We produce a concrete task for this: SketchTransfer, and we show that state-of-the-art domain transfer algorithms still struggle with this task. The state-of-the-art technique which achieves over 95\% on MNIST $\xrightarrow{}$ SVHN transfer only achieves 59\% accuracy on the SketchTransfer task, which is much better than random (11\% accuracy) but falls short of the 87\% accuracy of a classifier trained directly on labeled sketches. This indicates that this task is approachable with today's best methods but has substantial room for improvement.
BibTeX
@inproceedings{lamb2020sketchtransfer,
  author = {Alex Lamb and Sherjil Ozair and Vikas Verma and David Ha},
  title = {{SketchTransfer: A Challenging New Task for Exploring Detail-Invariance and the Abstractions Learned by Deep Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=8PPpBZcvOdj},
}
Figure