When we look at a hand-drawn sketch, we see a static image. But when we create a sketch, we engage in a rhythmic, sequential process. We start with a primary structure—a circle for a head or a long line for a horizon—and gradually layer in secondary details. Traditional computer vision often ignores this history, treating sketches as mere "black-and-white photos" composed of pixels.
However, a growing body of research argues that the true "DNA" of a sketch lies not in its final pixels, but in the temporal sequence of its strokes. By modeling sketches as time-series data rather than static grids, researchers are unlocking more robust recognition, more accurate 3D reconstruction, and a deeper understanding of human intent.
The Process is the Signal
The fundamental insight of this field is that the order in which strokes are drawn is not random; it reflects a structural hierarchy. As Pinaki Nath Chowdhury and colleagues observed in their work on FS-COCO, humans typically draw primary structural elements before secondary appendages. This temporal saliency provides a natural "roadmap" for machine learning models.
Early attempts to capture this, such as the Sketch-a-Net architecture by Timothy Hospedales and Yongxin Yang, represented sketches as multi-channel inputs where different channels encoded early, middle, and late stroke stages. This allowed the network to learn that a stroke's meaning often depends on when it appeared in the drawing process.
From Pixels to Vectors and Graphs
To fully leverage temporal information, the research has shifted away from raster images toward vector-based representations. Instead of a grid of intensities, a sketch is treated as a sequence of coordinate points $(x, y)$ and pen states (up/down).
- Recurrent and Transformer Models: Because sketches are sequences, they are natural candidates for architectures like LSTMs and Transformers. Leo Sampaio Ferraz Ribeiro and John Collomosse introduced Sketchformer, which uses Transformers to model long-range dependencies between strokes, outperforming models that treat sketches as static images.
- Graph Representations: Other researchers, like Lumin Yang, treat strokes as nodes in a graph. In SketchGCN, the model uses Graph Convolutional Networks to capture spatial proximity while maintaining the discrete identity of each stroke.
- Geometric Primitives: For highly structured tasks like character recognition, models like Stroke-based Character Reconstruction by Zhewei Huang deconstruct images into sequences of Bezier curves. This forces the model to ignore non-structural noise and focus on the "skeleton" of the character.
Resolving Ambiguity Through Time
Why does this temporal data matter so much? It resolves geometric ambiguities that are impossible to crack from a static 2D image.
In 3D modeling, Yulia Gryaditskaya demonstrated that the temporal order of strokes helps distinguish between intended 3D intersections and accidental 2D occlusions. By following the designer's "scaffolding" lines in order, their algorithm can lift a 2D sketch into a 3D wireframe with far greater plausibility. Similarly, in VRSKETCH2SHAPE, Yizi Chen showed that treating VR sketches as temporal sequences allows models to "hallucinate" complete 3D objects from only a few initial strokes.
Beyond Recognition: Explainability and Behavior
Treating sketches as sequences also makes AI more interpretable. Traditional saliency maps show which pixels a model looks at, but Hmrishav Bandyopadhyay has developed frameworks that explain model decisions at the stroke level. If a model identifies a "bicycle," it can point to the specific sequence of strokes that represent the wheels.
This temporal lens even extends to psychology. In VS-LLM, Xuchen Li and colleagues use the construction process of a drawing to assess depression, arguing that the effort and progression of the drawing—captured through stroke timing—are more diagnostic than the final image itself.
By moving from the "what" (the static image) to the "how" (the sequential process), this research direction aligns machine perception with the way humans actually create and understand the world through drawing.