A Neural Representation of Sketch Drawings

David Ha, Douglas Eck

Introduction

Researchers at Google Brain have introduced sketch-rnn, a Sequence-to-Sequence Variational Autoencoder (VAE) that learns to produce stroke-based vector drawings instead of pixel grids. By training on millions of human-drawn sketches, the model captures abstract concepts, allowing it to complete unfinished drawings, interpolate between different objects, and even perform "sketch arithmetic."

Examples of vector sketches

The Challenge of Vector Graphics

Most generative models for images, such as GANs or standard VAEs, operate on a grid of pixels. While effective for photos, this doesn't reflect how humans perceive or create art. We don't think in pixels; we think in abstract concepts and express them through a sequence of motor actions—strokes of a pen.

The challenge in modeling sketches as vector data (sequences of points) is that drawings are highly variable, often "noisy," and require the model to understand when to lift the pen or stop drawing entirely. Previous attempts at vector generation were often limited to specific domains like handwriting or simple characters. This paper seeks a robust, general-purpose framework for generating complex, multi-class vector drawings.

Architecture and Methods

The authors utilize a Sequence-to-Sequence Variational Autoencoder (VAE) architecture to process and generate these stroke sequences.

The sketch-rnn Sequence-to-Sequence VAE architecture
  • Encoder: A bidirectional RNN (specifically an LSTM) processes a sequence of strokes and compresses it into a latent vector $z$.
  • Decoder: An autoregressive RNN (using HyperLSTM for better sequence generation) takes the latent vector $z$ and predicts the next stroke.
  • Data Representation: Instead of absolute coordinates, each point is represented as a 5-element vector: $(\Delta x, \Delta y, p_1, p_2, p_3)$.
  • $(\Delta x, \Delta y)$ are offsets from the previous point.
  • $p_1, p_2, p_3$ are "one-hot" states indicating if the pen is touching the paper, lifted, or if the drawing is finished.
  • Output Distribution: The decoder predicts the next point using a Gaussian Mixture Model (GMM) for the offsets and a categorical distribution for the pen states.
  • Training & Sampling: The model uses a weighted loss function combining reconstruction loss ($L_R$) and KL divergence ($L_{KL}$). A "temperature" parameter $\tau$ is introduced during sampling to control the trade-off between following the model's most likely path (low $\tau$) and introducing creative randomness (high $\tau$).

The QuickDraw Dataset

The primary engine behind this work is the QuickDraw dataset. It contains 50 million vector drawings across hundreds of classes, collected from an online game where users had 20 seconds to draw an object. The authors simplified the strokes using the Ramer–Douglas–Peucker algorithm to keep sequences under 200 points. For this paper, they focused on 75 initial classes (e.g., cat, firetruck, yoga poses, owl) with 70,000 training samples per class.

Key Insights and Capabilities

  • Abstract Generalization: The model doesn't just memorize; it learns the concept. If fed a sketch of a cat with three eyes, the decoder (trained on normal cats) will often "correct" it to a two-eyed cat in its reconstruction.
  • Latent Space Arithmetic: Much like Word2Vec, the latent space $z$ allows for semantic analogies. For example: [Full Pig] - [Pig Head] + [Cat Head] = [Full Cat]. This demonstrates that the model encodes body parts as independent concepts within the vector space.
  • The Coherency Trade-off: There is a fundamental tension between reconstruction accuracy and latent space coherence. By adjusting the weight of the KL loss, researchers found that a higher weight forces the model to learn more "meaningful" global features (like "a body") rather than just local line segments.
  • Creative Completion: Because the decoder is autoregressive, it can function as a standalone "sketch completer," taking an unfinished human input and generating multiple plausible endings.
  • Interpolation and Robustness: The model can smoothly morph one sketch into another (e.g., a cat into a pig) through the latent space, producing coherent intermediate sketches. Furthermore, even when provided with sketches from untrained classes (like feeding a toothbrush into a cat-model), the model attempts to map the input's orientation and scale onto its learned features.

Limitations and Future Work

While sketch-rnn represents a significant step forward, it has a few limitations:

  • Sequence Length: The model struggles with very complex drawings exceeding 300 points, as training becomes significantly more difficult.
  • Large-Scale Multi-class Modeling: Currently, a single model cannot effectively represent the entire diversity of the QuickDraw dataset (all 75 classes simultaneously) without becoming incoherent. Future work might involve incorporating explicit class information outside the latent space.
  • Aesthetics and Cross-Domain Generation: Future iterations could incorporate user-rating data to steer the latent vector toward more "artistic" sketches. Additionally, combining this with pixel-based models (like Pix2Pix) could allow for generating photo-realistic images directly from these neural vector sketches.

Summary updated

Abstract

We present sketch-rnn, a recurrent neural network able to construct stroke-based drawings of common objects. The model is trained on a dataset of human-drawn images representing many different classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.
BibTeX
@inproceedings{ha2018neural,
  author = {David Ha and Douglas Eck},
  title = {{A Neural Representation of Sketch Drawings}},
  booktitle = {International Conference on Learning Representations},
  year = {2018},
  url = {https://openreview.net/forum?id=Hy6GHpkCW},
}
Figure