Self-Supervised Learning for Enhancing Spatial Awareness in Free-Hand Sketches

Xin Wang, Tengjie Li, Sicong Zang, Shikui Tu, Lei Xu

Abstract

Free-hand sketch, as a versatile medium of communication, can be viewed as a collection of strokes arranged in a spatial layout to convey a concept. Due to the abstract nature of the sketches, changes in stroke position may make them difficult to recognize. Recently, Graphic sketch representations are effective in representing sketches. However, existing methods overlook the significance of the spatial layout of strokes and the phenomenon of strokes being drawn in the wrong positions is common. Therefore, we developed a self-supervised task to correct stroke placement and investigate the impact of spatial layout on learning sketch representations. For this task, we propose a spatially aware method, named SketchGloc, utilizing multiple graphs for graphic sketch representations. This method utilizes grids for each stroke to describe the spatial layout with other strokes, allowing for the construction of multiple graphs. Unlike other methods that rely on a single graph, this design conveys more detailed spatial layout information and alleviates the impact of misplaced strokes. The experimental results demonstrate that our model outperforms existing methods in both our proposed task and the traditional controllable sketch synthesis task. Additionally, we found that SketchGloc can learn more robust representations under our proposed task setting.

Introduction

Free-hand sketching is a natural way for humans to communicate concepts through a sequence of strokes. However, these sketches are often abstract and "noisy," with strokes frequently misplaced due to speed, style, or physical constraints. While recent advances in Graph Neural Networks (GNNs) have improved sketch representation, they often lack spatial awareness—the ability to understand the relative positioning of strokes (e.g., eyes within a face) regardless of minor placement errors.

The Sketch Reorganization Task

To address this, the authors introduce a self-supervised pretext task called Sketch Reorganization. In this task, a sketch is intentionally disrupted by adding Gaussian noise to stroke positions (placement errors). The model must then learn to reconstruct the original, correctly organized sketch. This forces the network to develop a robust internal representation of the sketch's global structure and topological relationships.

The SketchGloc Architecture

The proposed model, SketchGloc, utilizes a novel multi-graph approach to capture detailed spatial relationships.

Architecture of SketchGloc featuring the Geometric Localization Module and Multi-Graph GCN.

The architecture consists of several key components:

  • Stroke Encoding: An RNN (LSTM) encoder processes individual strokes into initial embeddings.
  • Geometric Localization Module: For every pair of strokes, the model uses a 3x3 grid centered on one stroke to determine the relative position of the other. This captures topological relationships (inclusion, orientation) that are more invariant to noise than raw coordinates.
  • Multi-Graph GCN: Instead of a single adjacency matrix, SketchGloc constructs nine separate graphs corresponding to the nine grid cells. This allows the model to learn specific spatial weights for relationships like "above," "below," or "inside."
  • Reconstruction: A GCN encoder fuses these spatial features into a latent code, which an RNN decoder then uses to generate the final sequence of strokes.

Experimental Evaluation

SketchGloc was evaluated on the QuickDraw dataset across three subsets (DS1, DS2, DS3) designed to test generalization across categories with varying semantic gaps and structural similarities. Performance was measured using category recognition (Rec) and latent space retrieval (Ret).

Robustness to Noise

SketchGloc demonstrates remarkable robustness compared to existing models. In the reorganization task with 30% placement noise, SketchGloc achieved a 91.70% recognition rate on DS1. In contrast, the previous state-of-the-art (SP-gra2seq) dropped to 73.40%, and standard models like Sketch-RNN plummeted to 45.21%.

Qualitative comparison of SketchGloc against baseline models under varying levels of placement noise.

Qualitative Superiority

As shown in the qualitative results, SketchGloc ("ours") consistently produces recognizable and well-structured sketches even when the input is highly fragmented. Other models often fail to maintain the global topology, resulting in "exploded" or unrecognizable outputs. This proves that spatial awareness helps the model maintain the "big picture" even when local visual cues are disrupted.

Conclusion

By incorporating a 3x3 grid-based multi-graph representation, SketchGloc effectively captures the spatial layout of free-hand sketches. The self-supervised reorganization task further enhances its ability to correct placement errors, leading to more robust and accurate sketch synthesis and recognition. Future work may explore finer grid granularities for complex sketches and rotation-invariant representations to handle sketches drawn at different angles.

Summary updated

Abstract

Free-hand sketch, as a versatile medium of communication, can be viewed as a collection of strokes arranged in a spatial layout to convey a concept. Due to the abstract nature of the sketches, changes in stroke position may make them difficult to recognize. Recently, Graphic sketch representations are effective in representing sketches. However, existing methods overlook the significance of the spatial layout of strokes and the phenomenon of strokes being drawn in the wrong positions is common. Therefore, we developed a self-supervised task to correct stroke placement and investigate the impact of spatial layout on learning sketch representations. For this task, we propose a spatially aware method, named SketchGloc, utilizing multiple graphs for graphic sketch representations. This method utilizes grids for each stroke to describe the spatial layout with other strokes, allowing for the construction of multiple graphs. Unlike other methods that rely on a single graph, this design conveys more detailed spatial layout information and alleviates the impact of misplaced strokes. The experimental results demonstrate that our model outperforms existing methods in both our proposed task and the traditional controllable sketch synthesis task. Additionally, we found that SketchGloc can learn more robust representations under our proposed task setting. The source code is available at https://github.com/CMACH508/SketchGloc.
BibTeX
@inproceedings{wang2024self,
  author = {Xin Wang and Tengjie Li and Sicong Zang and Shikui Tu and Lei Xu},
  title = {{Self-Supervised Learning for Enhancing Spatial Awareness in Free-Hand Sketches}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=ynkv1IDHOU},
}
Figure