AI-Sketcher : A Deep Generative Model for Producing High-Quality Sketches

Nan Cao, Xin Yan, Yang Shi, Chaoran ChenAAAI 2019 · DOI

Introduction

Generating sketches—sequences of pen strokes—presents a unique challenge for AI compared to standard raster image generation. While models like Sketch-RNN have made significant strides by treating drawing as a sequential language problem, they suffer from two primary flaws: spatial blindness and class confusion. Because these models focus primarily on the sequence of strokes, they often lose track of global geometry, leading to distorted drawings where features like eyes or noses are misaligned. Furthermore, they struggle to distinguish between multiple categories, often producing incoherent hybrids when trained on diverse datasets.

AI-Sketcher is a hybrid deep generative model designed to overcome these limitations. By combining a Convolutional Neural Network (CNN)-based spatial autoencoder with a Recurrent Neural Network (RNN)-based sequential model, it achieves significantly higher structural integrity and controllability.

Methods

AI-Sketcher builds upon the Variational Autoencoder (VAE) framework with three critical architectural innovations:

  1. CNN-based Autoencoder: To address spatial blindness, the authors integrated a CNN that processes a rasterized version of the sketch. This allows the model to capture the global spatial layout, ensuring that individual strokes maintain their proper relative positions.
  2. The Influence Layer: Unlike standard RNNs that rely only on the final hidden state, AI-Sketcher’s influence layer acts as a global memory mechanism. It looks back at all previous hidden states to guide the generation of each new stroke based on the entire drawing history.
  3. Conditional Vector: To support multi-class generation, a "one-hot" conditional vector is injected into the encoder and decoder. This allows the model to switch between specific categories (e.g., different facial expressions) within a single unified model.
Architecture comparison between (a) Sketch-RNN and (b) the proposed AI-Sketcher, featuring (c) the CNN-based autoencoder details.

Datasets and Evaluation

The model was evaluated using two primary datasets: QuickDraw, containing 50 million doodles across 75 categories, and FaceX, a specialized dataset of 5 million high-quality facial expression sketches. These datasets provided a rigorous testing ground for the model's ability to handle both simple objects and complex, multi-part human features.

Key Results and Insights

AI-Sketcher demonstrates a clear improvement in both qualitative and quantitative metrics:

  • Structural Accuracy: Qualitative comparisons show that while baseline models often produce "melted" or distorted features, AI-Sketcher maintains clear, recognizable structures. The combination of the CNN for layout and the RNN for stroke sequence prevents the "drift" common in purely sequential models.
  • Superior Quality: In a user study, AI-Sketcher received an average quality rating of 3.9/5, significantly outperforming Sketch-RNN.
  • Latent Space Organization: t-SNE visualizations revealed that AI-Sketcher successfully organizes different categories into tight, coherent clusters, whereas previous models exhibited significant overlap and noise.
  • Efficiency: Despite its architectural complexity, the model remains highly efficient, generating individual strokes in approximately 0.013 seconds on standard hardware.
Qualitative comparison of sketches generated by different model configurations across various emotional categories, demonstrating the impact of the Influence Layer and Autoencoder.

Conclusion

AI-Sketcher represents a significant leap forward in AI-assisted drawing. By treating sketches as both sequences and spatial layouts, the model achieves a level of structural integrity previously unattainable. While the model currently requires a rasterization preprocessing step and carries a higher training cost, its ability to produce high-quality, category-specific sketches opens new doors for professional design assistance and creative prototyping.

Summary updated

BibTeX
@inproceedings{cao2019ai,
  author = {Nan Cao and Xin Yan and Yang Shi and Chaoran Chen},
  title = {{AI-Sketcher : A Deep Generative Model for Producing High-Quality Sketches}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33012564},
}

Loading related content...

Figure