Parallel Stroke Unmasking as Sketch Abstraction, Trained Without Human Drawings

Teaching a computer to match human sketches to photos needs paired sketches, and people are slow to draw them. The Sketchy dataset alone took about 4,000 drawing hours (Edge Augmentation for Large-Scale Sketch Recognition without Sketches). The usual workaround is to make fake sketches from photos. Raw edge maps make poor fakes, though, because they keep every texture line that a person would leave out. The best photo-only result on fine-grained shoe retrieval comes from Learning Deep Sketch Abstraction. That paper trains a reinforcement-learning agent to decide, one stroke at a time, whether to keep or drop each stroke of an edge drawing. It reaches 21.17% top-1 accuracy on the QMUL-Shoe-V2 retrieval benchmark, against 8.86% for earlier synthetic sketches and 34.38% when the retrieval model is trained on real human sketches.

This note proposes a different way to make fake sketches. First, break photo edges into strokes and replace each stroke with the nearest entry in a vocabulary of stroke shapes learned from real human drawings. Then train a masked transformer that fills in the whole set of strokes in parallel over a few rounds. If generation stops after $k$ strokes, the result is an abstraction of the full drawing, so a single model yields sketches at many levels of detail. It needs no reward-driven training and no stroke-by-stroke decoding, and because sampling is parallel it can produce many varied sketches per photo cheaply. The measurable claim is that retrieval models trained on these sketches beat 21.17% on QMUL-Shoe-V2 while still using zero human sketch–photo pairs.

Core Claim

Two ingredients carry the claim, and each can be tested on its own.

The first is that rounding strokes to a human vocabulary removes much of the gap between synthetic and real sketches. VQ-SGen showed that human strokes can be split into a shape code and a location code. Each code is an entry in a learned table of examples, which is what vector quantization means here. If the shape table is learned from unpaired human strokes, for example from all QuickDraw categories, then rounding an edge stroke to its nearest shape code replaces a machine-exact contour with a human-like one. Prior work had to fake human wobble by hand-designed means: the random perturbations of Deep Sketch Abstraction, and the dilation and dropout of Deep Plastic Surgery. Here the wobble comes from real human strokes.

The second is that the order in which strokes are filled in can serve as the abstraction itself. MaskGIT generates images by starting from an all-masked grid of tokens. Each round it predicts every masked position and keeps the most confident predictions, finishing in 8–12 rounds on a cosine schedule. We apply the same procedure to an unordered set of stroke tokens conditioned on the photo. Strokes that the photo and the strokes already placed make easy to predict get committed first.

Confidence alone may favor long outer contours over the small details, such as laces or toe caps, that tell one shoe from another. So the score for keeping a stroke adds a term for how much it helps match the drawing to its own photo:

$$s_i = \log p_\theta(c_i \mid \text{photo}, \text{committed strokes}) + \lambda\, \Delta R_i .$$

Here $c_i$ is the predicted token for stroke slot $i$, and $p_\theta$ is the masked transformer's probability for it. $\Delta R_i$ is the gain in matching score between the partial sketch and its source photo when stroke $i$ is added. That score comes from a frozen retrieval encoder trained only on full synthetic sketches. $\lambda$ trades off the two terms. This plays the same role as the RL agent's recognizability reward, but it is applied at sampling time and costs no extra training.

Research Program

Building the training sketches. Shoe and chair photos are turned into edge maps using the recipe in Edge Augmentation: an ensemble of three edge detectors, thinning to one-pixel lines, and random binarization thresholds. We follow it closely because that paper shows the thinning step matters. The thinned edges are then traced into polyline strokes. Optionally, the background can be removed first with a saliency mask, as in CoGS.

Tokenizing strokes. A two-table stroke tokenizer in the style of VQ-SGen is trained on unpaired human strokes. Each traced edge stroke becomes a pair of tokens, one for shape and one for location.

Training the generator. A bidirectional transformer without positional order, so that the strokes form an unordered set as in StrokeFusion, learns to recover masked stroke tokens. It conditions on a photo encoder. Mask ratios are drawn from a cosine schedule, and empty slots carry an "absent" token so that drawings can have different numbers of strokes.

Sampling. Generation runs from a fully masked set of slots and stops at a stroke budget $k$. That budget is swept from 30% to 100% of the stroke count of the full edge drawing.

Downstream use. The resulting multi-level, multi-sample abstractions train a fine-grained retrieval model. That model is then tested on real human sketches.

The analysis question running through all of this is whether the order of parallel filling recovers human stroke saliency. On the human test sketches, which come with recorded drawing order, we measure how well our commit order correlates with that human order. Deep Sketch Abstraction's keep/skip ranking gets the same measurement.

Baselines

The naive reference point is training the retrieval model on the full set of traced edge strokes, with no abstraction and no vocabulary rounding. This reproduces the "edges as sketches" setting that Deep Sketch Abstraction reports as 8.86% for earlier synthesis.

The strongest existing competitor is the reinforcement-learning stroke-pruning agent of Learning Deep Sketch Abstraction, at 21.17% top-1 on QMUL-Shoe-V2 using photo-derived sketches only. It is the strongest competitor because it is the only method in the file that addresses this exact setting: abstraction for retrieval, without human pairs. We report its published number. We also re-run its agent on our own traced edge strokes, so that a win cannot come from a better edge source. Both its agent and our sampler get the same tuning budget, and the retrieval network, triplet protocol and image resolution are held fixed. Beating it means exceeding 21.17%, and our re-run of it, by more than the seed-to-seed spread over three seeds. Closing a substantial part of the gap to the 34.38% human-sketch upper bound would be the strong result. Photo Pre-Training, But for Sketch reports 50.75% acc@1, but it uses human sketches. It sets a ceiling, not a competitor.

A raster baseline trains the same retrieval backbone directly on the edge images from the first step. This is the edge-augmentation recipe, which reaches 70.6% on the PACS image-to-sketch recognition benchmark. Its fine-grained retrieval result is not reported, so we produce it.

A cheap heuristic baseline keeps the longest $k$ strokes, or a random $k$ strokes, from the rounded set. This tests whether learned ordering matters at all.

Three ablations isolate the ingredients. The first swaps in a stroke-shape table learned from edge strokes in place of the human-learned one; this isolates the rounding claim. The second sets $\lambda = 0$, leaving pure confidence ordering. The third is an autoregressive decoder over the same tokens, in the VQ-SGen style, which generates strokes one after another and truncates at $k$. It isolates parallel filling against sequential decoding in both quality and wall-clock time.

Why not just…?

Why not just run the RL abstraction agent and be done? It is the main threat, and it is run directly, so the comparison settles the question. Its known weaknesses are these: it needs a separately trained policy for each brevity setting $\delta$, it decides stroke by stroke, and its hand-written perturbations are the only thing making edges look human. If our sampler matches it only once $\lambda > 0$, the honest conclusion is that the gain comes from recognizability guidance plus vocabulary rounding, not from parallel decoding as such. We would report it that way.

Why not just train on randomized thin edges as rasters? Edge augmentation is very strong for category recognition (70.6% PACS). But Photo-Sketching shows that edge-style targets keep texture lines that people drop: a pix2pix model on edges scores contour F1 0.536, against 0.822 for its human-style model. Fine-grained retrieval is where that clutter should hurt most. If the raster baseline matches us, the vector machinery is not justified for retrieval, and we would say so.

Why not continuous stroke diffusion like StrokeFusion or SketchKnitter? SketchKnitter fixes drawings at 96 points, and StrokeFusion was only trained on human vector datasets. Neither offers a natural "stop after $k$ strokes, in order of importance" operation. Continuous latents also cannot be snapped to a human vocabulary, which is the first ingredient. The ablation with an edge-learned stroke table, and an optional run of StrokeFusion-style latent diffusion on the same synthetic data, would show whether discreteness matters.

Why not keep the longest strokes? If length ordering comes close to learned ordering, the masked model adds little beyond the rounding step. This is a cheap baseline and is run directly.

Proposed Experiments

The primary benchmark is QMUL-Shoe-V2 top-1 accuracy, with retrieval models trained only on synthetic sketches from the benchmark's photos. QMUL-Chair-V2 serves as a second category; the file reports no number for chairs under synthetic-only training.

Secondary measurements cover four things. The first is classifier recognizability of abstractions at 40% stroke removal, compared against the 70.4% reported by Deep Sketch Abstraction under its protocol. The second is Fréchet distance between our sketches and held-out human sketches, as a measure of realism. The third is rank correlation between our commit order and human drawing order. The fourth is sampling time against both the autoregressive ablation and the RL agent.

Three checks are decisive. The full system must beat 21.17% without human pairs. The human-learned stroke table must beat the edge-learned one at equal stroke budgets; otherwise the rounding claim fails. And the confidence-only sampler ($\lambda = 0$) must beat keeping the longest strokes; otherwise parallel unmasking does not recover saliency on its own.