SketchSegNet+: An End-to-End Learning of RNN for Multi-Class Sketch Semantic Segmentation

Yonggang Qi, Zheng-Hua TanIEEE Access 2019 · DOI · OpenReview

Abstract

Sketch semantic segmentation is the task of identifying and labeling individual components within a free-hand sketch. This paper presents SketchSegNet+, an end-to-end recurrent neural network (RNN) framework designed to solve the "where is what" problem in human sketches. By treating sketches as sequences of pen strokes rather than static images, the model captures the temporal "grammar" of drawing. Utilizing a massive new dataset of 150,000 annotated sketches, SketchSegNet+ significantly outperforms previous methods in multi-class, fine-grained sketch understanding.

Introduction

While computer vision has excelled at recognizing objects in photographs, sketches remain a significant challenge due to their abstract nature, lack of color/texture, and high variability. Most existing models treat a sketch as a single object label. However, advanced applications like sketch-to-image synthesis or intelligent graphic design tools require a deeper understanding of the parts of a sketch (e.g., distinguishing the wings from the body of an airplane).

SketchSegNet+ addresses "stroke-level semantic segmentation" by automatically assigning labels to every stroke. Unlike previous category-specific or template-reliant methods, this framework provides a flexible, multi-class system that understands the context of how humans draw.

Methodology

The authors frame sketch segmentation as a sequence-to-sequence (Seq2Seq) problem. The model translates a sequence of pen coordinates into a sequence of part labels.

  1. Encoder: A Bidirectional Long Short-Term Memory (BiLSTM) network processes strokes in both forward and backward order to capture global context.
  2. Latent Space: The encoder compresses the sketch into a 128-dimensional latent vector representing the drawing's essence.
  3. Decoder: A unidirectional LSTM predicts the semantic label for each point based on the latent vector and preceding stroke information.
  4. Data Augmentation: To train the large-scale model, the authors manually labeled a small subset and used a generative model (Sketch-RNN) to produce 150,000 synthetic variations, preserving drawing order and semantic labels.

Datasets

The study introduces SketchSeg-150K, a large-scale dataset containing 150,000 sketches across 20 categories with 57 distinct part labels, derived from Google’s QuickDraw. The model was also benchmarked against the SPG Dataset (20,000 sketches, 25 categories).

Results and Discussion

SketchSegNet+ demonstrates superior performance over state-of-the-art models like Edge-PG and USPG. On the SketchSeg-150K dataset, it achieved a Probabilistic Rand Index (PRI) of 0.97, compared to 0.75 and 0.69 for competitors. The model reached approximately 89% stroke-based accuracy (SA) and 87% component-based accuracy (CA).

Semantic segmentation results for various object categories including airplanes, traffic lights, and envelopes.

Qualitatively, the model excels at maintaining "part integrity," rarely mislabeling segments of a single component. It successfully groups strokes that are spatially distant—such as two separate eyes—by understanding their shared semantic meaning.

Segmentation results for complex categories such as faces, angels, and birds, demonstrating the model's ability to handle intricate part relationships.

The model's performance remains robust across diverse categories, though accuracy slightly decreases for objects with many small, overlapping parts, such as complex angels or airplanes.

Conclusion

SketchSegNet+ provides a robust foundation for fine-grained sketch understanding. By capturing the temporal context of human drawing through an RNN architecture, it enables machines to interpret the semantic structure of sketches. This capability is a critical step toward more intuitive human-computer interaction in creative design and search applications.

Summary updated

Abstract

We investigate the problem of stroke-level sketch segmentation, which is to automatically assign strokes of a given sketch with semantic labels. Solving the problem of sketch segmentation opens the door for fine-grained sketch interpretation, which can benefit many novel sketch-based applications, including sketch recognition and sketch-based image retrieval. In this paper, we propose an approach for multi-class sketch semantic segmentation by considering it as a sequence-to-sequence generation problem. Specifically, an end-to-end learned network <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SketchSegNet</i> +, built on recurrent neural networks (RNN), is presented to translate a sequence of strokes into a sequence of semantic labels. In addition, a large-scale stroke-level sketch segmentation dataset is constructed for the first time, which is composed of 150K annotated free-hand human sketch selected from <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">QuickDraw</i> . The dataset will be released publicly. The experimental results of stroke-level sketch semantic segmentation on this novel dataset and the SPG dataset demonstrate the effectiveness of our approach.
BibTeX
@article{qi2019sketchsegnet,
  author = {Yonggang Qi and Zheng-Hua Tan},
  title = {{SketchSegNet+: An End-to-End Learning of RNN for Multi-Class Sketch Semantic Segmentation}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2929804},
}
Figure