Vector graphics (images stored as editable curves and colored shapes rather than pixels) can be made in two ways. The slow way runs hundreds of gradient steps per image. Each step renders the curves, scores the render with a large frozen vision model, and nudges the curve parameters. This gives high quality but takes 5 to 30 minutes per output. The fast way trains a network to emit the curves in one pass. That is hundreds of times faster, but the network breaks on inputs unlike its training data and smooths away fine detail. We propose a middle path built around one specific change. The fast network is never treated as the final answer. Instead it is trained, end to end, to produce a starting point that a small, fixed number of optimization steps (one to ten) can polish best. The two parts are separate, swappable modules that share a clean interface: a "proposer" that guesses the geometry and a "refiner" that runs a bounded optimizer on it.
Put in two sentences: today's fast vector generators are trained to be right on the first try, so when they are wrong nothing cheap can fix them. We train the generator for what it will look like after a few steps of test-time correction, which should keep most of the speed of one-pass generation while recovering much of the robustness of per-image optimization, especially on unfamiliar inputs.
Core Claim
Our central hypothesis is about how the amortized model is trained. A proposer trained through a few unrolled refinement steps will, at the same wall-clock budget, reach better fidelity than an equally sized proposer trained only for one-pass accuracy and then refined with the same steps. We expect the gap to be largest on out-of-distribution inputs. Unrolled means the training loss is computed after the refiner's steps, and gradients flow back through those steps.
Formally, let $c$ be the condition (an input image or a text prompt), let $G_\theta(c)$ be the proposer's output (control points, stroke widths, colors, layer order), and let $R_k(x; c)$ be $k$ steps of gradient descent on a test-time objective $L(x; c)$ through a differentiable rasterizer. We train
$$\min_\theta \; \mathbb{E}_{c,\;k \sim p(k)} \Big[ \mathcal{L}_{\text{task}}\big(R_k(G_\theta(c); c), c\big) \Big],$$
where $\mathcal{L}_{\text{task}}$ is the training-time quality loss (distance to a teacher's output plus a perceptual term on the render), and $p(k)$ is a distribution over small budgets that includes $k=0$. Sampling $k$ at train time is meant to give one model that works at any budget the user chooses, instead of one model per budget.
Positioning Against Prior Work
The motivating evidence comes from the two extremes of vector synthesis. VectorFusion optimizes SVG paths by backpropagating score-distillation gradients (gradients from a frozen pixel diffusion model) through a differentiable rasterizer. It reaches 81.3% OpenCLIP H/14 R-Precision but takes 10–20 minutes per SVG. It already warm-starts from a Stable Diffusion sample traced into vectors, which shows that initialization matters, but that initialization was never trained to be refined.
At the fast end, SwiftSketch distills a ~10-minute optimization teacher into a 0.5-second diffusion model over Bezier control points. It reports that quality degrades badly outside its 15 training categories and that its learned refinement network over-smooths fine contours. SVGBuilder assembles icons from a rigid library of normalized components in about 1.36 seconds, but it cannot deform shapes to fit subtle geometry. StrokeNUWA tokenizes strokes and fixes broken endpoints with a rule-based post-processor at ~19 seconds per icon.
All three fast systems fix their errors, if at all, with either a learned one-shot refiner or hand-written rules. None uses objective-driven refinement, and none trains its generator with refinement in mind. That pairing is exactly what we add.
The method borrows from two sources. Amortized Proximal Optimization learns optimizer-side quantities by differentiating through a one-step lookahead and shows this can stay stable in 16-bit training. We apply the same lookahead idea to learning initializations for a geometric optimizer rather than preconditioners for network training. Amortized Latent Steering shows the opposite design choice in language models: it replaces iterative test-time optimization entirely with a fixed offline correction, and its authors concede this is blunt across heterogeneous inputs. We treat that as a constraint on design. Full amortization loses adaptivity, which is why we keep a small amount of genuine per-instance optimization.
The modular-design literature supplies the interface discipline. FaceX-Zoo separates backbones from loss heads with factory patterns so either can be swapped. The Universal Causal Evaluation Engine enforces a fixed boundary between data generators and estimators. We impose the same kind of boundary between proposer and refiner. It takes the form of a typed geometry state with a fixed set of path parameters, so any proposer can feed any refiner. This makes "is the benefit specific to one refiner objective?" a testable question rather than an architectural accident.
The file also warns that abstraction layers can add latency. We will therefore measure the interface's overhead directly rather than assume it is free.
Research Program
We will build the system in three parts.
The proposer–refiner interface. This is a geometry state holding Bezier control points, widths, RGBA fills, and layer order. Two proposers plug into it: a SwiftSketch-style continuous diffusion decoder, and an SVGBuilder-style component assembler whose components are relaxed to have deformable control points. Refiners plug in by implementing a step function on that state. We will implement three refiners: CLIP-feature matching to an input image, LPIPS (a learned perceptual distance) to an input image, and score distillation for text prompts.
Refinement-aware training. We train the proposer with the objective above. Exact unrolling through score distillation is costly, because its gradient involves a diffusion network's forward pass at every step. We will therefore compare three gradient estimators on cost and accuracy:
- full unrolling, for $k \le 5$ with image-matching refiners;
- a first-order approximation that ignores second derivatives through the refiner;
- a truncated scheme that unrolls only the final step.
Which estimator holds up is itself a finding.
Characterizing the frontier. We will measure the latency–fidelity frontier for every proposer and refiner pairing, with budgets spanning 0 to 25 steps. This produces the benchmark that the file identifies as missing: a principled account of where hybrid methods sit between 0.5 seconds and 10 minutes.
Baselines
The naive reference point is the unmodified one-pass generator with no refinement. For image-to-sketch this is SwiftSketch, reported at 0.95 CLIP Top-1 accuracy on in-distribution categories at ~0.5 seconds. For text-to-SVG it is SVGBuilder, reported at 15.93 FID on ColorSVG-100K at ~1.36 seconds. We will retrain both on our data splits so numbers are comparable, and report the published figures alongside.
The strongest quality reference is full per-instance optimization. ControlSketch is reported at 0.97 CLIP Top-1 at ~10 minutes in SwiftSketch. VectorFusion reports 81.3% OpenCLIP H/14 R-Precision at 10–20 minutes. These set the ceiling we try to approach. We do not expect to match them at a fraction of the cost.
The key ablation, and the real competitor, is naive warm-starting. It uses the identical proposer architecture trained only for one-pass accuracy, then refined at test time by the same refiner for the same $k$ steps. It gets the same data, the same parameter count, the same total training compute (we will give it the extra GPU-hours that unrolling costs, spent on longer training), and the same hyperparameter search budget, including a tuned refiner step size.
A second ablation replaces the objective-driven refiner with a learned one-step refinement network, as in SwiftSketch. This tests whether genuine optimization matters at all. A third starts the bounded refiner from a raster image traced into vectors, which is a cheaper version of VectorFusion's warm start.
Beating the naive warm-start means a statistically reliable gain in CLIP Top-1 or R-Precision, plus human preference, at matched wall-clock time on held-out categories. It also means a cost of no more than about one point of CLIP Top-1 at $k=0$ in-distribution.
No reported result was found on Lacuna for SwiftSketch's out-of-distribution accuracy, nor for any few-step refinement baseline. Those numbers will be produced here.
Why not just…?
Why not just warm-start the existing fast model with a few optimization steps? This is the most serious threat, and it is exactly our main ablation. If an off-the-shelf proposer refined for five steps matches our refinement-aware proposer at equal time, the core claim fails, and the contribution shrinks to a benchmark of hybrid budgets. There is a reason to expect a gap. A one-pass-trained model is pushed to commit to averaged, smoothed geometry, which SwiftSketch's over-smoothing reflects, and a few gradient steps may not escape that. A refinement-trained proposer can instead place strokes where the refiner's gradients are informative. The out-of-distribution split is where we expect this to show. If it does not appear there, we will report a negative result.
Why not just train the one-pass student on more diverse data? SwiftSketch's failures outside its training distribution are partly a data problem. More teacher data from more categories would help both the baseline and our method. We will run a data-scaling control by training the plain student on two and four times as many teacher samples. This tells us whether refinement-aware training buys the equivalent of a fixed multiple of data. Even if extra data closes the in-distribution gap, per-instance refinement remains the only component that sees the actual test input's objective. So we predict the advantage persists on categories that no amount of category-limited data covers.
Why not just use a better learned refinement network? A learned refiner is still amortized, so it inherits the generalization limits of its training data. SwiftSketch's refiner is documented to over-smooth facial contours. Our second ablation compares a learned refiner against $k$ objective-driven steps at matched latency on held-out categories.
Why not trace a fast raster diffusion sample and refine that? VectorFusion reports that SD + LIVE tracing reaches only 65.6% R-Precision versus 81.3% for full optimization, and that tracing produces fragmented, hard-to-edit paths. Our third ablation applies the same bounded refiner to traced initializations to see whether a few steps can close that gap.
Proposed Experiments
Testbed A: image to vector sketch. We regenerate teacher pairs in the style of SwiftSketch's 35,000-sample, 15-class synthetic dataset. Five classes are held out entirely, and we add multi-object scenes as a harder out-of-distribution set. Metrics are CLIP Top-1 recognition, LPIPS to the input, human preference, and wall-clock latency on a single GPU, including the interface overhead.
Testbed B: text to colored SVG. This uses ColorSVG-100K. Metrics are FID (comparable to SVGBuilder's 15.93), OpenCLIP H/14 R-Precision (comparable to VectorFusion's protocol), and latency.
The primary output is the frontier of fidelity against seconds for each method at budgets of 0, 1, 2, 5, 10, and 25 steps.
Three checks are decisive:
- Main comparison. At matched wall-clock time, refinement-aware proposers beat naive warm-starting, with the margin growing on held-out categories.
- Cost at zero steps. We measure how much performance is lost when refinement-aware training is used without any refinement. If the loss is large, the method is a budget-specific trade rather than an anytime improvement.
- Modularity test. We train with the CLIP refiner and test with the LPIPS or score-distillation refiner. If the gains transfer, the proposer has learned objective-agnostic "refinability" and the modular interface earns its keep. If they vanish, proposer and refiner must be co-trained, which tells practitioners the boundary is not a free swap.
We will also report gradient-estimator ablations, comparing full unrolling, first-order, and truncated schemes on accuracy and training cost. These determine whether the approach scales to score-distillation refiners at all.
References
- VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models
- SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation
- SVGBuilder: Component-Based Colored SVG Generation with Text-Guided Autoregressive Transformers
- StrokeNUWA—Tokenizing Strokes for Vector Graphic Synthesis
- Amortized Proximal Optimization
- Amortized Latent Steering: Low-Cost Alternative to Test-Time Optimization
- FaceX-Zoo: A PyTorch Toolbox for Face Recognition
- Universal Causal Evaluation Engine: An API for empirically evaluating causal inference models