Abstract
Sketch semantic segmentation is the task of identifying and labeling individual components within a free-hand sketch. This paper presents SketchSegNet+, an end-to-end recurrent neural network (RNN) framework designed to solve the "where is what" problem in human sketches. By treating sketches as sequences of pen strokes rather than static images, the model captures the temporal "grammar" of drawing. Utilizing a massive new dataset of 150,000 annotated sketches, SketchSegNet+ significantly outperforms previous methods in multi-class, fine-grained sketch understanding.
Introduction
While computer vision has excelled at recognizing objects in photographs, sketches remain a significant challenge due to their abstract nature, lack of color/texture, and high variability. Most existing models treat a sketch as a single object label. However, advanced applications like sketch-to-image synthesis or intelligent graphic design tools require a deeper understanding of the parts of a sketch (e.g., distinguishing the wings from the body of an airplane).
SketchSegNet+ addresses "stroke-level semantic segmentation" by automatically assigning labels to every stroke. Unlike previous category-specific or template-reliant methods, this framework provides a flexible, multi-class system that understands the context of how humans draw.
Methodology
The authors frame sketch segmentation as a sequence-to-sequence (Seq2Seq) problem. The model translates a sequence of pen coordinates into a sequence of part labels.
- Encoder: A Bidirectional Long Short-Term Memory (BiLSTM) network processes strokes in both forward and backward order to capture global context.
- Latent Space: The encoder compresses the sketch into a 128-dimensional latent vector representing the drawing's essence.
- Decoder: A unidirectional LSTM predicts the semantic label for each point based on the latent vector and preceding stroke information.
- Data Augmentation: To train the large-scale model, the authors manually labeled a small subset and used a generative model (Sketch-RNN) to produce 150,000 synthetic variations, preserving drawing order and semantic labels.
Datasets
The study introduces SketchSeg-150K, a large-scale dataset containing 150,000 sketches across 20 categories with 57 distinct part labels, derived from Google’s QuickDraw. The model was also benchmarked against the SPG Dataset (20,000 sketches, 25 categories).
Results and Discussion
SketchSegNet+ demonstrates superior performance over state-of-the-art models like Edge-PG and USPG. On the SketchSeg-150K dataset, it achieved a Probabilistic Rand Index (PRI) of 0.97, compared to 0.75 and 0.69 for competitors. The model reached approximately 89% stroke-based accuracy (SA) and 87% component-based accuracy (CA).
Qualitatively, the model excels at maintaining "part integrity," rarely mislabeling segments of a single component. It successfully groups strokes that are spatially distant—such as two separate eyes—by understanding their shared semantic meaning.
The model's performance remains robust across diverse categories, though accuracy slightly decreases for objects with many small, overlapping parts, such as complex angels or airplanes.
Conclusion
SketchSegNet+ provides a robust foundation for fine-grained sketch understanding. By capturing the temporal context of human drawing through an RNN architecture, it enables machines to interpret the semantic structure of sketches. This capability is a critical step toward more intuitive human-computer interaction in creative design and search applications.
Summary updated