SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks

Lumin Yang, Jiajie Zhuang, Hongbo Fu, Xiangzhi Wei, Kun Zhou, Youyi ZhengACM Trans. Graphics 2021 · OpenReview

TL;DR

This paper introduces SketchGNN, a novel graph neural network (GNN) approach for automatically segmenting and labeling freehand sketches. By representing sketches as graphs and employing a unique two-branch architecture with static and dynamic convolutions, SketchGNN achieves state-of-the-art accuracy while being significantly more parameter-efficient than existing methods.

What Problem Does This Paper Address?

Interpreting freehand sketches is challenging for computers due to their inherent ambiguity, varying abstraction levels, and artistic styles. While previous methods have focused on segmenting entire sketches or analyzing them as images or sequences, this paper tackles the problem of semantic sketch segmentation, which involves identifying and labeling individual parts of sketched objects. Existing deep learning methods either treat sketches as images, ignoring crucial stroke structure, or as sequences, neglecting point proximity. This gap limits their effectiveness in applications requiring detailed understanding of sketch components.

Methods

SketchGNN treats a vector sketch as a graph where nodes are sampled points along strokes and edges represent stroke structure. The core innovation lies in its two-branch architecture:

  • Static Branch: Uses static graph convolutions to capture intra-stroke features, focusing on the inherent structure within individual strokes.
  • Dynamic Branch: Employs dynamic graph convolutions that adaptively update graph connectivity layer by layer, allowing for the capture of inter-stroke relationships and proximity information. This branch also incorporates a Mix Pooling Block that aggregates features at three levels: point-level, stroke-level (via a novel "stroke pooling" operation), and sketch-level.

This multi-level feature extraction is crucial for understanding the complex relationships within sketches. The paper also highlights the use of absolute coordinates for node features, which naturally encodes proximity, unlike sequence-based methods.

Datasets

The authors evaluated SketchGNN on four publicly available sketch datasets:

  • SPG dataset: A large-scale dataset built upon QuickDraw, with 25 categories and 800 sketches per category.
  • SketchSeg-150K dataset: Also from QuickDraw, but simpler with fewer semantic labels per category (2-4) and approximately 150,000 sketches.
  • Huang14 dataset: A smaller dataset with 30 sketches per category.
  • TU-Berlin dataset: Another smaller dataset with 80 sketches per category.

For the Huang14 and TU-Berlin datasets, synthetic training data was generated by rendering 3D models and extracting edge maps to approximate sketches.

Key Insights and Ideas

  • Graph Representation is Key: Treating sketches as graphs effectively captures both stroke structure and point proximity, which are crucial for accurate semantic segmentation.
  • Static + Dynamic is Better: Combining static graph convolutions (for intra-stroke details) with dynamic graph convolutions (for inter-stroke relationships) provides a more comprehensive feature representation.
  • Three-Level Features Matter: Aggregating features at point, stroke, and sketch levels, particularly with the novel stroke pooling, significantly improves segmentation consistency and accuracy.
  • Parameter Efficiency: SketchGNN achieves state-of-the-art results with orders of magnitude fewer parameters than existing methods, making it suitable for resource-constrained applications.

Key Results

SketchGNN demonstrated significant improvements over existing methods on the SPG dataset, achieving 11.2% higher pixel-based accuracy and 18.2% higher component-based accuracy compared to the previous best method (FastSeg+GC). The model also showed strong generalization capabilities on other datasets. Notably, SketchGNN's parameter count is around 434KB, which is two orders of magnitude smaller than image-based (40.9MB) and sequence-based (23.4MB) methods.

Limitations and Future Work

The authors acknowledge several limitations:

  • Ambiguity in Sketches: The inherent ambiguity in freehand sketches regarding part position and shape can lead to mislabeling.
  • Domain Gap: Differences between training and testing data (e.g., synthetic vs. real sketches, different viewing angles) can impact performance.
  • High-Level Semantics: The current graph representation primarily uses node positions and proximity, lacking awareness of higher-level semantic constraints (e.g., a face having only one nose).

Future work suggested includes:

  • Incorporating stroke order, direction, and spatial relations into the graph representation.
  • Exploring more flexible graph structures.
  • Utilizing recurrent modules for learning intact graph representations.
  • Reshaping the architecture for scene-level sketch segmentation and sketch recognition tasks.

Summary updated

Abstract

We introduce SketchGNN, a convolutional graph neural network for semantic segmentation and labeling of freehand vector sketches. We treat an input stroke-based sketch as a graph with nodes representing the sampled points along input strokes and edges encoding the stroke structure information. To predict the per-node labels, our SketchGNN uses graph convolution and a static-dynamic branching network architecture to extract the features at three levels, i.e., point-level, stroke-level, and sketch-level. SketchGNN significantly improves the accuracy of the state-of-the-art methods for semantic sketch segmentation (by 11.2% in the pixel-based metric and 18.2% in the component-based metric over a large-scale challenging SPG dataset) and has magnitudes fewer parameters than both image-based and sequence-based methods.
BibTeX
@article{yang2021sketchgnn,
  author = {Lumin Yang and Jiajie Zhuang and Hongbo Fu and Xiangzhi Wei and Kun Zhou and Youyi Zheng},
  title = {{SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks}},
  journal = {ACM Trans. Graphics},
  year = {2021},
  url = {https://openreview.net/forum?id=3mCNJbiHsQy},
}
Figure