On Training Sketch Recognizers for New Domains

Kemal Tugrul Yesilbek, T. Metin SezginWorkshops @ CVPR 2021 · DOI

Abstract

Sketch recognition models often struggle when transitioning from controlled laboratory settings to real-world, high-pressure environments. This paper investigates this "reality gap" using a real-time strategy (RTS) game as a testbed. By comparing traditional machine learning methods with modern deep learning architectures in small-data regimes, the authors demonstrate that traditional classifiers like SVMs significantly outperform CNNs. Furthermore, the study reveals that data collected in-situ is fundamentally different from data collected in lab settings, posing a major challenge for deploying sketch-based interfaces in niche domains.

Introduction

Most progress in sketch recognition has been driven by massive datasets like Google’s QuickDraw. However, developing specialized applications—such as military strategy games or medical tools—requires a custom visual vocabulary for which massive datasets do not exist. Developers face two primary hurdles: ecological validity (ensuring lab-collected data matches real-world usage) and data sufficiency (training models with very few examples).

Methodology

The researchers used the RTS game OpenRA to explore these challenges. They designed a custom visual vocabulary for game actions and units.

The custom visual vocabulary designed for the OpenRA testbed, featuring both visual-alike and semi-syntactic symbols across construction, production, and commanding categories.

To study the "reality gap," three data collection protocols were established:

  • In-Lab (I-L): Participants drew symbols on a tablet in a relaxed environment.
  • Pseudo-Realistic (P-R): Participants drew under time pressure with scoring to mimic game urgency.
  • Realistic (R): Data was collected during actual gameplay using a "Wizard of Oz" setup, where a human labeled sketches in real-time to maintain a seamless player experience.

The study compared traditional classifiers (SVMs and Gradient Boosting Machines) against various CNN architectures and embedding-based models pre-trained on QuickDraw.

The Reality Gap and Model Performance

The results highlighted a brutal "dataset shift." Models trained on clean "In-Lab" data saw a massive performance drop when tested on "Realistic" gameplay data. For instance, a Large CNN (CNN-L) that achieved 91.7% accuracy on lab data dropped to 83.3% when faced with real-world sketches. This suggests that the cognitive load of an actual task fundamentally alters sketching behavior in ways that simple timers (Pseudo-Realistic) cannot replicate.

Comprehensive accuracy table showing model performance across different training and testing sets. Traditional methods like SVM consistently outperform deep learning models in the Realistic (R) testing scenarios.

Surprisingly, in this "small data" regime (roughly 50-120 examples per class), traditional methods proved far more robust. On the realistic test set, a standard SVM achieved 98.7% accuracy, while the best CNN only reached 91.7%. Even with transfer learning and 50x data augmentation, the best deep learning model (95.1%) could not surpass the baseline SVM.

Conclusion and Future Work

The primary barrier to building robust sketch recognizers for new domains is the high cost of collecting realistic, in-situ data. The authors suggest that future research should focus on Style Transfer. By learning to transform clean, lab-collected sketches into the "messy" style of high-pressure gameplay, it may be possible to synthetically generate the volume of data required for deep learning models to reach their full potential in specialized applications.

Summary updated

BibTeX
@inproceedings{yesilbek2021training,
  author = {Kemal Tugrul Yesilbek and T. Metin Sezgin},
  title = {{On Training Sketch Recognizers for New Domains}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  doi = {10.1109/cvprw53098.2021.00243},
}
Figure