Modeling Sketches as Temporal Stroke Sequences

This direction is split into other directions: temporal information loss (temporal stroke dynamics). Its last version stays readable and citable here.

Version 1 of 1108 papers150 conceptsfrozen 2026-10-04

When we look at a hand-drawn sketch, we see a static image. But when we create a sketch, we engage in a rhythmic, sequential process. We start with a primary structure—a circle for a head or a long line for a horizon—and gradually layer in secondary details. Traditional computer vision often ignores this history, treating sketches as mere "black-and-white photos" composed of pixels.

However, a growing body of research argues that the true "DNA" of a sketch lies not in its final pixels, but in the temporal sequence of its strokes. By modeling sketches as time-series data rather than static grids, researchers are unlocking more robust recognition, more accurate 3D reconstruction, and a deeper understanding of human intent.

The Process is the Signal

The fundamental insight of this field is that the order in which strokes are drawn is not random; it reflects a structural hierarchy. As Pinaki Nath Chowdhury and colleagues observed in their work on FS-COCO, humans typically draw primary structural elements before secondary appendages. This temporal saliency provides a natural "roadmap" for machine learning models.

Early attempts to capture this, such as the Sketch-a-Net architecture by Timothy Hospedales and Yongxin Yang, represented sketches as multi-channel inputs where different channels encoded early, middle, and late stroke stages. This allowed the network to learn that a stroke's meaning often depends on when it appeared in the drawing process.

From Pixels to Vectors and Graphs

To fully leverage temporal information, the research has shifted away from raster images toward vector-based representations. Instead of a grid of intensities, a sketch is treated as a sequence of coordinate points $(x, y)$ and pen states (up/down).

  1. Recurrent and Transformer Models: Because sketches are sequences, they are natural candidates for architectures like LSTMs and Transformers. Leo Sampaio Ferraz Ribeiro and John Collomosse introduced Sketchformer, which uses Transformers to model long-range dependencies between strokes, outperforming models that treat sketches as static images.
  2. Graph Representations: Other researchers, like Lumin Yang, treat strokes as nodes in a graph. In SketchGCN, the model uses Graph Convolutional Networks to capture spatial proximity while maintaining the discrete identity of each stroke.
  3. Geometric Primitives: For highly structured tasks like character recognition, models like Stroke-based Character Reconstruction by Zhewei Huang deconstruct images into sequences of Bezier curves. This forces the model to ignore non-structural noise and focus on the "skeleton" of the character.

Resolving Ambiguity Through Time

Why does this temporal data matter so much? It resolves geometric ambiguities that are impossible to crack from a static 2D image.

In 3D modeling, Yulia Gryaditskaya demonstrated that the temporal order of strokes helps distinguish between intended 3D intersections and accidental 2D occlusions. By following the designer's "scaffolding" lines in order, their algorithm can lift a 2D sketch into a 3D wireframe with far greater plausibility. Similarly, in VRSKETCH2SHAPE, Yizi Chen showed that treating VR sketches as temporal sequences allows models to "hallucinate" complete 3D objects from only a few initial strokes.

Beyond Recognition: Explainability and Behavior

Treating sketches as sequences also makes AI more interpretable. Traditional saliency maps show which pixels a model looks at, but Hmrishav Bandyopadhyay has developed frameworks that explain model decisions at the stroke level. If a model identifies a "bicycle," it can point to the specific sequence of strokes that represent the wheels.

This temporal lens even extends to psychology. In VS-LLM, Xuchen Li and colleagues use the construction process of a drawing to assess depression, arguing that the effort and progression of the drawing—captured through stroke timing—are more diagnostic than the final image itself.

By moving from the "what" (the static image) to the "how" (the sequential process), this research direction aligns machine perception with the way humans actually create and understand the world through drawing.

Papers in this version

Showing 60 of 108 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_modeling_sketches_as_temporal_stroke_seq_1a5c9467,
  title = {Modeling Sketches as Temporal Stroke Sequences},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-sketches-as-temporal-stroke-sequences/rdr_1a5c9467543947b5aca34002275ba640},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_modeling_sketches_as_temporal_stroke_seq_1a5c9467_v1,
  title = {Modeling Sketches as Temporal Stroke Sequences},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-sketches-as-temporal-stroke-sequences/rdr_1a5c9467543947b5aca34002275ba640/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}