Introduction
Generating high-quality Scalable Vector Graphics (SVG) has long been a challenge for artificial intelligence. While Large Language Models (LLMs) excel at writing code, they often treat SVG syntax as a purely text-based programming language. This unimodal approach lacks a visual feedback loop; a single misplaced coordinate in a text string can lead to "catastrophic failures," such as facial features rendered outside the bounds of a head. Furthermore, the scarcity of high-quality, clean SVG datasets compared to the billions of available text-image pairs has hindered the development of models that can maintain structural integrity and generalize to complex prompts.
The DuetSVG Framework
To address these limitations, researchers have introduced DuetSVG, a unified multimodal model built on the Janus-Pro architecture. The core innovation of DuetSVG is its ability to "imagine" a raster image before writing the corresponding vector code. By jointly generating image tokens and SVG tokens, the model provides itself with internal visual guidance, resulting in SVGs that are significantly more semantically accurate and geometrically coherent.
Methodology
DuetSVG employs a unified multimodal sequence where the model first outputs image tokens followed by SVG text tokens. This sequence ensures that the generated image tokens act as a "mental map" for the subsequent vector generation. The training process occurs in two distinct stages:
- Stage 1 (T2I Pretraining): The model is pretrained on massive text-to-image datasets to learn strong visual and semantic priors.
- Stage 2 (Multi-task SFT): The model is fine-tuned on a mix of text-to-image, text-to-SVG, and image-to-SVG tasks to align these different modalities.
During inference, the model utilizes test-time scaling with SVG resampling. It generates several raster image candidates, selects the best one via a CLIP-based verifier, and then generates the SVG in chunks. Each chunk is rendered and verified using the LPIPS (Learned Perceptual Image Patch Similarity) metric to ensure the vector paths remain consistent with the chosen guidance image.
Data and Benchmarking
Recognizing that existing SVG datasets are often noisy, the authors introduced SVG-Hub-1M and SVG-Hub-5M, curated datasets containing up to 5 million high-quality SVGs. These datasets were enhanced with multi-level captioning—ranging from short prompts to detailed annotations—to improve the model's semantic understanding. Evaluation on the SVG-Hub-5M and SArena-Icon benchmarks demonstrated that DuetSVG consistently outperforms state-of-the-art models like StarVector and LLM4SVG.
Key Results and Versatility
DuetSVG exhibits superior performance across a variety of tasks, including text-to-SVG generation, image-to-SVG conversion, and complex SVG editing. Unlike traditional vectorization tools that produce a "spaghetti" of paths, DuetSVG maintains logical path semantics, grouping tokens for specific objects together.
As shown in the examples above, the model can successfully complete partial SVGs and follow instructions to replace specific elements (e.g., replacing a pen with a badge or a hat with headphones) while maintaining the style of the original graphic.
Limitations and Future Work
While DuetSVG represents a significant leap forward, it still faces challenges with extremely intricate textures and subtle color gradients due to fixed vision encoder resolutions. Future work may involve implementing dynamic high-resolution strategies to capture finer details and further refining the handling of complex overlapping geometries to eliminate occasional rendering artifacts.
Summary updated