Abstract
Deep networks have achieved excellent results in perceptual tasks, yet their ability to generalize to variations not seen during training has come under increasing scrutiny. In this work, we focus on their ability to have invariance towards the presence or absence of details. For example, humans are able to watch cartoons, which are missing many visual details, without being explicitly trained to do so. The failure of machine learning algorithms to do this indicates a significant gap in generalization between human abilities and the abilities of deep networks. We propose a dataset, SketchTransfer, to study the detail-invariance problem concretely. We show that state-of-the-art domain transfer algorithms still struggle with this task, achieving 59% accuracy compared to the 87% accuracy of a classifier trained directly on labeled sketches. This indicates that this task is approachable with today's best methods but has substantial room for improvement.
Introduction: The Detail-Invariance Problem
Humans possess a remarkable ability called detail-invariance. We can recognize a "dog" whether it is a high-resolution photograph, a cartoon, or a 20-second "QuickDraw" doodle. We achieve this by learning high-level abstractions—like the specific shape of a cat's ears—while discarding irrelevant textures.
In contrast, deep networks are often "texture-biased," relying on local patterns (like fur or skin texture) rather than global structure. When these details are removed, model performance collapses. The SketchTransfer task quantifies how well models can transfer knowledge from a detail-rich domain (real photos) to a detail-sparse domain (sketches).
The SketchTransfer Dataset
The SketchTransfer dataset creates a correspondence between two existing benchmarks to test cross-domain abstraction:
- Source Domain (Real Images): CIFAR-10, consisting of 60,000 32x32 color images across 10 classes.
- Target Domain (Sketches): QuickDraw, a collection of millions of human-drawn sketches.
The authors mapped specific QuickDraw categories to CIFAR-10 classes (e.g., mapping "Police Car" to "Automobile"). Because QuickDraw lacks a "Deer" category, the final SketchTransfer test set contains 9 classes.
Methods and Evaluation
The study evaluates several state-of-the-art Unsupervised Domain Adaptation (UDA) and self-supervised learning techniques to bridge the "abstraction gap":
- Adversarial Domain Adaptation: Using a discriminator to align internal representations of real images and sketches.
- Virtual Mixup Training (VMT): A consistency-based approach that interpolates between images and pseudo-labels to smooth decision boundaries.
- Rotation Prediction: A self-supervised task forcing the model to learn structural features by predicting image orientation.
- CyCada / CycleGAN: Generative models that translate real images into the style of sketches.
- Dirt-T: An iterative refinement procedure that enforces the "cluster assumption" on target data.
Key Results
The results highlight a significant performance gap between current UDA methods and supervised learning:
| Method | Accuracy |
|---|---|
| Random Guessing | 11% |
| Source Only (CIFAR-10 only) | ~23.3% |
| **Best UDA (VMT + Adv. |
- DIRT-T) | ~58.9%** | | Supervised (Trained on labeled sketches) | ~87.3% |
Standard models trained only on CIFAR-10 fail significantly (~23% accuracy), confirming a heavy reliance on color and texture. While UDA methods improve performance to nearly 59%, they still lag 28% behind supervised models, indicating that current transfer learning struggles with extreme changes in the level of detail.
Insights and Future Work
Nearest-neighbor analysis shows that while raw pixel-space comparisons are useless for sketches, models trained with domain adaptation begin to group sketches by semantic meaning. However, models still struggle to identify "highly salient" but visually small features (like ear shapes) that humans use to distinguish classes.
Future work may explore "intermediate" domains, such as cartoons, to serve as a stepping stone for abstraction. Additionally, the authors suggest exploring Artist Agents (models that learn to "draw" stroke-by-stroke) or Meta-Learning to force models to understand fundamental structure rather than surface textures.
Summary updated