DuetSVG: Unified Multimodal SVG Generation with Internal Visual Guidance

Peiying Zhang, Nanxuan Zhao, Matthew Fisher, Yiran Xu, Jing Liao, Difan Liu

Introduction

Generating high-quality Scalable Vector Graphics (SVG) has long been a challenge for artificial intelligence. While Large Language Models (LLMs) excel at writing code, they often treat SVG syntax as a purely text-based programming language. This unimodal approach lacks a visual feedback loop; a single misplaced coordinate in a text string can lead to "catastrophic failures," such as facial features rendered outside the bounds of a head. Furthermore, the scarcity of high-quality, clean SVG datasets compared to the billions of available text-image pairs has hindered the development of models that can maintain structural integrity and generalize to complex prompts.

The DuetSVG Framework

To address these limitations, researchers have introduced DuetSVG, a unified multimodal model built on the Janus-Pro architecture. The core innovation of DuetSVG is its ability to "imagine" a raster image before writing the corresponding vector code. By jointly generating image tokens and SVG tokens, the model provides itself with internal visual guidance, resulting in SVGs that are significantly more semantically accurate and geometrically coherent.

The DuetSVG architecture utilizes a unified autoregressive transformer to process prompt tokens, image tokens, and SVG tokens through specialized heads.

Methodology

DuetSVG employs a unified multimodal sequence where the model first outputs image tokens followed by SVG text tokens. This sequence ensures that the generated image tokens act as a "mental map" for the subsequent vector generation. The training process occurs in two distinct stages:

  1. Stage 1 (T2I Pretraining): The model is pretrained on massive text-to-image datasets to learn strong visual and semantic priors.
  2. Stage 2 (Multi-task SFT): The model is fine-tuned on a mix of text-to-image, text-to-SVG, and image-to-SVG tasks to align these different modalities.

During inference, the model utilizes test-time scaling with SVG resampling. It generates several raster image candidates, selects the best one via a CLIP-based verifier, and then generates the SVG in chunks. Each chunk is rendered and verified using the LPIPS (Learned Perceptual Image Patch Similarity) metric to ensure the vector paths remain consistent with the chosen guidance image.

Data and Benchmarking

Recognizing that existing SVG datasets are often noisy, the authors introduced SVG-Hub-1M and SVG-Hub-5M, curated datasets containing up to 5 million high-quality SVGs. These datasets were enhanced with multi-level captioning—ranging from short prompts to detailed annotations—to improve the model's semantic understanding. Evaluation on the SVG-Hub-5M and SArena-Icon benchmarks demonstrated that DuetSVG consistently outperforms state-of-the-art models like StarVector and LLM4SVG.

Key Results and Versatility

DuetSVG exhibits superior performance across a variety of tasks, including text-to-SVG generation, image-to-SVG conversion, and complex SVG editing. Unlike traditional vectorization tools that produce a "spaghetti" of paths, DuetSVG maintains logical path semantics, grouping tokens for specific objects together.

DuetSVG demonstrates versatility across text-to-SVG, image-to-SVG, and instruction-based SVG editing tasks.

As shown in the examples above, the model can successfully complete partial SVGs and follow instructions to replace specific elements (e.g., replacing a pen with a badge or a hat with headphones) while maintaining the style of the original graphic.

Limitations and Future Work

While DuetSVG represents a significant leap forward, it still faces challenges with extremely intricate textures and subtle color gradients due to fixed vision encoder resolutions. Future work may involve implementing dynamic high-resolution strategies to capture finer details and further refining the handling of complex overlapping geometries to eliminate occasional rendering artifacts.

Summary updated

BibTeX
@inproceedings{zhang2026duetsvg,
  author = {Peiying Zhang and Nanxuan Zhao and Matthew Fisher and Yiran Xu and Jing Liao and Difan Liu},
  title = {{DuetSVG: Unified Multimodal SVG Generation with Internal Visual Guidance}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=37B6huY5nP},
}
Figure