TL;DR
This paper introduces SketchGNN, a novel graph neural network (GNN) approach for automatically segmenting and labeling freehand sketches. By representing sketches as graphs and employing a unique two-branch architecture with static and dynamic convolutions, SketchGNN achieves state-of-the-art accuracy while being significantly more parameter-efficient than existing methods.
What Problem Does This Paper Address?
Interpreting freehand sketches is challenging for computers due to their inherent ambiguity, varying abstraction levels, and artistic styles. While previous methods have focused on segmenting entire sketches or analyzing them as images or sequences, this paper tackles the problem of semantic sketch segmentation, which involves identifying and labeling individual parts of sketched objects. Existing deep learning methods either treat sketches as images, ignoring crucial stroke structure, or as sequences, neglecting point proximity. This gap limits their effectiveness in applications requiring detailed understanding of sketch components.
Methods
SketchGNN treats a vector sketch as a graph where nodes are sampled points along strokes and edges represent stroke structure. The core innovation lies in its two-branch architecture:
- Static Branch: Uses static graph convolutions to capture intra-stroke features, focusing on the inherent structure within individual strokes.
- Dynamic Branch: Employs dynamic graph convolutions that adaptively update graph connectivity layer by layer, allowing for the capture of inter-stroke relationships and proximity information. This branch also incorporates a Mix Pooling Block that aggregates features at three levels: point-level, stroke-level (via a novel "stroke pooling" operation), and sketch-level.
This multi-level feature extraction is crucial for understanding the complex relationships within sketches. The paper also highlights the use of absolute coordinates for node features, which naturally encodes proximity, unlike sequence-based methods.
Datasets
The authors evaluated SketchGNN on four publicly available sketch datasets:
- SPG dataset: A large-scale dataset built upon QuickDraw, with 25 categories and 800 sketches per category.
- SketchSeg-150K dataset: Also from QuickDraw, but simpler with fewer semantic labels per category (2-4) and approximately 150,000 sketches.
- Huang14 dataset: A smaller dataset with 30 sketches per category.
- TU-Berlin dataset: Another smaller dataset with 80 sketches per category.
For the Huang14 and TU-Berlin datasets, synthetic training data was generated by rendering 3D models and extracting edge maps to approximate sketches.
Key Insights and Ideas
- Graph Representation is Key: Treating sketches as graphs effectively captures both stroke structure and point proximity, which are crucial for accurate semantic segmentation.
- Static + Dynamic is Better: Combining static graph convolutions (for intra-stroke details) with dynamic graph convolutions (for inter-stroke relationships) provides a more comprehensive feature representation.
- Three-Level Features Matter: Aggregating features at point, stroke, and sketch levels, particularly with the novel stroke pooling, significantly improves segmentation consistency and accuracy.
- Parameter Efficiency: SketchGNN achieves state-of-the-art results with orders of magnitude fewer parameters than existing methods, making it suitable for resource-constrained applications.
Key Results
SketchGNN demonstrated significant improvements over existing methods on the SPG dataset, achieving 11.2% higher pixel-based accuracy and 18.2% higher component-based accuracy compared to the previous best method (FastSeg+GC). The model also showed strong generalization capabilities on other datasets. Notably, SketchGNN's parameter count is around 434KB, which is two orders of magnitude smaller than image-based (40.9MB) and sequence-based (23.4MB) methods.
Limitations and Future Work
The authors acknowledge several limitations:
- Ambiguity in Sketches: The inherent ambiguity in freehand sketches regarding part position and shape can lead to mislabeling.
- Domain Gap: Differences between training and testing data (e.g., synthetic vs. real sketches, different viewing angles) can impact performance.
- High-Level Semantics: The current graph representation primarily uses node positions and proximity, lacking awareness of higher-level semantic constraints (e.g., a face having only one nose).
Future work suggested includes:
- Incorporating stroke order, direction, and spatial relations into the graph representation.
- Exploring more flexible graph structures.
- Utilizing recurrent modules for learning intact graph representations.
- Reshaping the architecture for scene-level sketch segmentation and sketch recognition tasks.
Summary updated