A Neural Representation of Sketch Drawings

David Ha, Douglas Eck

Abstract

We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.

Introduction

Traditional generative models for images typically treat pictures as static grids of pixels. However, human drawing is inherently sequential and abstract. We do not perceive the world as a rasterized grid; rather, we communicate through sequences of strokes. The challenge addressed in this work is the representation and generation of vector images—sequences of pen movements—rather than static pixel arrays. This requires a model capable of handling variable-length sequences, understanding the intent behind each stroke, and maintaining global coherence across the entire drawing.

Methods

The proposed model, sketch-rnn, is a Sequence-to-Sequence Variational Autoencoder (VAE) designed specifically for vector data.

Model Architecture

The architecture consists of two primary components:

  1. The Encoder: A bidirectional RNN that processes an input sequence of pen strokes and compresses them into a latent vector ($z$). This vector captures the high-level "concept" of the drawing.
  2. The Decoder: An autoregressive RNN (a HyperLSTM) that uses the latent vector to predict the next stroke in the sequence.
The Sketch-RNN architecture, featuring a bidirectional RNN encoder and a HyperLSTM decoder.

Data Representation

Instead of pixels, each data point is represented as a 5-element vector $(\Delta x, \Delta y, p_1, p_2, p_3)$. Here, $(\Delta x, \Delta y)$ are the offsets from the previous point, while $(p_1, p_2, p_3)$ represent binary pen states: "pen down" (drawing), "pen up" (moving), and "end of drawing." The decoder outputs the parameters of a Gaussian Mixture Model (GMM) for the offsets, allowing for diverse, non-deterministic generation.

Dataset

The model was trained on the QuickDraw dataset, focusing on 75 categories such as "cat," "firetruck," and "yoga pose." Each class contains 70,000 training samples. To facilitate learning, the Ramer-Douglas-Peucker algorithm was used to simplify complex human strokes into fewer points.

Results and Insights

Sketch-RNN demonstrates several unique capabilities that distinguish it from pixel-based models.

Concept over Precision

A key finding is the model's ability to prioritize conceptual coherence over literal reconstruction. When fed a "wrong" image (e.g., a cat with three eyes), the latent space regularization often forces the decoder to "correct" the input, producing a standard two-eyed cat. This suggests the model has learned the underlying structure of the object class.

Sketch Algebra and Interpolation

Because the model maps drawings to a continuous latent space, it enables "sketch algebra." For instance, subtracting the latent vector of a "pig head" from a "full pig" and adding a "cat head" can result in a "full cat." Furthermore, the model can smoothly interpolate between different drawings, creating coherent intermediate forms.

Latent space interpolations and variations across multiple categories including firetrucks, yoga poses, and owls.

Coherency and Temperature

The authors found that increasing the weight of the Kullback-Leibler (KL) divergence loss encourages a smoother latent space, leading to more "human-like" outputs. Additionally, a "temperature" parameter ($\tau$) allows users to control the diversity of the generated samples; lower temperatures produce standard sketches, while higher temperatures result in more creative, albeit chaotic, variations.

Discussion and Future Work

While Sketch-RNN excels at simple objects, it faces limitations with high-complexity drawings. Sequences exceeding 300 points often lead to over-smoothed or blurred results. Additionally, while the model can learn multiple classes, training a single model on a very large number of diverse classes simultaneously remains a challenge.

Future work involves incorporating explicit class labels to better organize the latent space and exploring creative applications. Sketch-RNN could serve as a "creative assistant" for artists, suggesting ways to complete a drawing or providing variations for design patterns.

Summary updated

Abstract

We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.
BibTeX
@inproceedings{ha2018neural,
  author = {David Ha and Douglas Eck},
  title = {{A Neural Representation of Sketch Drawings}},
  booktitle = {International Conference on Learning Representations},
  year = {2018},
  url = {https://openreview.net/forum?id=Hy6GHpkCW},
}
Figure