SDGraph: Multi-Level Sketch Representation Learning by Sparse-Dense Graph Architecture

Xi Cheng, Pingfa Feng, Zhichao Liao, Mingyu Fan, Long Zeng

Abstract: Freehand sketches exhibit unique sparsity and abstraction, necessitating learning pipelines distinct from those designed for images. For sketch learning methods, the central objective is to fully exploit the effective information embedded in sketches. However, there is limited research on what constitutes effective sketch information, which in turn constrains the performance of existing approaches. To tackle this issue, we first proposed the Multi-Level Sketch Representation Scheme to systematically identify the effective information. The scheme organizes sketch representation into three levels: sketch-level, stroke-level, and point-level. This design is based on the granularity of analytical elements, from coarse (sketch-level) to fine (point-level), thereby ensuring more comprehensive coverage of the sketch information. For each level, we conducted theoretical analyses and experimental evaluations to identify and validate the effective information. Building on the above studies, we developed SDGraph, a deep learning architecture designed to exploit the identified effective information across the three levels. SDGraph comprises two complementary modules: a Sparse Graph that treats strokes as nodes for sketch-level and stroke-level representation learning, and a Dense Graph that treats points as nodes for sketch-level and point-level representation learning. Both modules employ graph convolution along with down-sampling and up-sampling operations, enabling them to function as both encoder and decoder. Besides that, an information fusion module bridges the two graphs to further enhance feature extraction. SDGraph supports a wide range of sketch-related downstream tasks, achieving accuracy improvements of 1.15% and 1.70% over the state-of-the-art in classification and retrieval, respectively, and 36.58% improvement in vector sketch generation quality.

Introduction

Freehand sketches are fundamentally different from natural images. They are sparse, abstract, and typically represented as vector data—sequences of points and strokes—rather than dense pixel grids. While deep learning has revolutionized image processing, sketch learning requires specialized pipelines that can exploit the unique structural and temporal information inherent in drawings. A central challenge in the field is identifying and utilizing "effective information" within a sketch. Many existing models focus narrowly on global shapes or point sequences, often neglecting the complex structural relationships between individual strokes.

The Multi-Level Sketch Representation Scheme

To systematically address the information gap, the authors propose a Multi-Level Sketch Representation Scheme. This scheme categorizes sketch data into three granularities:

  1. Sketch-level: Captures the global structure and overall semantic meaning of the drawing.
  2. Stroke-level: Focuses on intra-stroke features (shape, length) and inter-stroke relations (spatial layout, parallelism, intersections).
  3. Point-level: Examines local geometric cues and the temporal order of points within a single stroke.
The Multi-Level Sketch Representation Scheme, identifying effective information at the sketch, stroke, and point levels.

Through theoretical and experimental analysis, the researchers identified which types of information significantly impact model performance. A key finding was that while intra-stroke temporal information (the direction a line is drawn) is highly effective, inter-stroke temporal information (the order in which different parts of an object are drawn) has a negligible impact on representation learning.

SDGraph Architecture

Building on this representation scheme, the authors developed SDGraph, a dual-module architecture designed to capture information across all three levels. SDGraph consists of two complementary graph-based components:

  • Sparse Graph (SGraph): This module treats individual strokes as nodes. By operating at this coarser granularity, SGraph efficiently learns the structural relationships between strokes without the computational burden of processing every individual point.
  • Dense Graph (DGraph): This module treats every point as a node, capturing fine-grained local geometry and intersections that the SGraph might miss.
Overview of the SDGraph pipeline, showing the multi-level representation, the Sparse-Dense graph architecture, and downstream tasks.

An Information Fusion Module bridges these two graphs, allowing for bidirectional feature exchange. Stroke-level context is passed down to points, while point-level details are aggregated back up to strokes, ensuring a holistic understanding of the sketch. The architecture follows an encoder-decoder structure, making it versatile for both discriminative and generative tasks.

Experimental Results

The model was evaluated on the QuickDraw dataset for classification and generation, and the Sketchy dataset for Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR).

  • Classification and Retrieval: SDGraph achieved state-of-the-art (SOTA) results, with a 75.37% accuracy on QuickDraw (a 1.15% improvement over previous benchmarks) and a 1.70% improvement in fine-grained retrieval accuracy.
  • Sketch Generation: By integrating SDGraph as the noise-prediction engine within a Denoising Diffusion Probabilistic Model (DDPM), the researchers achieved a massive 36.58% improvement in generation quality (FID score). Qualitatively, the generated sketches exhibit superior structural coherence and more accurate proportions compared to traditional RNN-based methods like SketchRNN.

Conclusion

SDGraph demonstrates that a multi-level, dual-scale approach is essential for mastering the nuances of freehand sketches. By distinguishing between effective and ineffective information and fusing sparse and dense representations, the model sets a new standard for sketch-related tasks. Future work will focus on reducing the computational overhead associated with varying sketch sizes and exploring even more granular data partitioning methods.

Summary updated

Abstract

Freehand sketches exhibit unique sparsity and abstraction, necessitating learning pipelines distinct from those designed for images. For sketch learning methods, the central objective is to fully exploit the effective information embedded in sketches. However, there is limited research on what constitutes effective sketch information, which in turn constrains the performance of existing approaches. To tackle this issue, we first proposed the Multi-Level Sketch Representation Scheme to systematically identify the effective information. The scheme organizes sketch representation into three levels: sketch-level, stroke-level, and point-level. This design is based on the granularity of analytical elements, from coarse (sketch-level) to fine (point-level), thereby ensuring more comprehensive coverage of the sketch information. For each level, we conducted theoretical analyses and experimental evaluations to identify and validate the effective information. Building on the above studies, we developed SDGraph, a deep learning architecture designed to exploit the identified effective information across the three levels. SDGraph comprises two complementary modules: a Sparse Graph that treats strokes as nodes for sketch-level and stroke-level representation learning, and a Dense Graph that treats points as nodes for sketch-level and point-level representation learning. Both modules employ graph convolution along with down-sampling and up-sampling operations, enabling them to function as both encoder and decoder. Besides that, an information fusion module bridges the two graphs to further enhance feature extraction. SDGraph supports a wide range of sketch-related downstream tasks, achieving accuracy improvements of 1.15\% and 1.70\% over the state-of-the-art in classification and retrieval, respectively, and 36.58\% improvement in vector sketch generation quality.
BibTeX
@misc{cheng2025sdgraph,
  author = {Xi Cheng and Pingfa Feng and Zhichao Liao and Mingyu Fan and Long Zeng},
  title = {{SDGraph: Multi-Level Sketch Representation Learning by Sparse-Dense Graph Architecture}},
  year = {2025},
  eprint = {2510.12192},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.12192},
}
Figure