SVGBuilder: Component-Based Colored SVG Generation with Text-Guided Autoregressive Transformers

Zehao Chen, Rong PanAAAI 2025 · DOI

Introduction

Generating high-quality, colored Scalable Vector Graphics (SVGs) from text prompts has long been a challenge for generative AI. Unlike raster images, which are composed of pixels, SVGs are defined by mathematical paths and XML code. This structural complexity often leads to syntax errors in Large Language Models (LLMs) or extreme latency in optimization-based methods.

SVGBuilder introduces a novel framework that treats SVGs as assemblies of discrete "components" rather than raw code strings. By utilizing a new 100,000-image dataset (ColorSVG-100K) and an autoregressive transformer, the model achieves a 604x speedup over traditional optimization methods while significantly outperforming state-of-the-art models like GPT-4 in visual fidelity.

The Challenge of Vector Graphics

Current approaches to SVG generation generally fall into two categories, both of which face significant limitations:

  1. Optimization-based methods: These start with random paths and iteratively refine them to match a target description. While they produce high-quality results, they are computationally expensive, often requiring over 20 minutes to generate a single icon.
  2. Language Model (LLM) methods: Models like GPT-4 attempt to write SVG code character-by-character. Because SVG syntax is verbose, these models frequently "hallucinate" invalid code or lose track of global structure, resulting in broken or messy graphics.

Furthermore, existing SVG datasets are often small or restricted to black-and-white strokes, leaving a gap in the research for vibrant, multi-colored vector art.

The Component-Based Approach

SVGBuilder addresses these issues by mimicking the workflow of human designers—assembling icons from a library of reusable shapes.

Overview of the SVGBuilder pipeline: from raw code to component library and final construction.

Methodology

  • Component Library Construction: The researchers decomposed thousands of SVGs into individual paths, which were then normalized (centered and scaled). Using Jaccard similarity, they merged nearly identical shapes into a consolidated "library."
  • The SVGBuilder Architecture: The model employs a multi-modal transformer. A Text Encoder processes the user's prompt, while an Image Encoder provides visual feedback. As the model generates the SVG component-by-component, the partial result is rasterized and fed back into the system, allowing the model to "see" where to place the next piece.
  • SVG Decoder: Instead of predicting raw XML, the GPT-2-based decoder predicts a sequence of seven tokens for each component: [Component ID, Offset X, Offset Y, Scale, R, G, B]. This ensures that every generated graphic is syntactically valid.

ColorSVG-100K Dataset

To support this research, the authors introduced ColorSVG-100K, the first large-scale dataset specifically for colored SVG generation. It contains 100,000 high-quality graphics. To improve the model's understanding of text-to-visual mappings, they used a BLIP-2 caption generator to augment the dataset with rich, descriptive metadata (e.g., transforming a simple "flower" label into "A bouquet of flowers and a pink ribbon").

Key Results and Insights

SVGBuilder demonstrates that predicting "Component IDs" instead of raw XML strings bypasses the syntax errors common in LLM-based generation. By stripping color and scale from the library components, the model learns the concept of a shape (like a "circle") once and treats size and color as independent parameters.

  • Incredible Speed: SVGBuilder generates an icon in roughly 1.36 seconds. In comparison, VectorFusion takes approximately 13.69 minutes.
  • Superior Quality: On the ColorSVG-100K test set, SVGBuilder achieved an FID score of 15.93, significantly better than GPT-4 (31.85) and IconShop (41.03).
  • Stability: Unlike other language-based models that struggle as the number of paths increases, SVGBuilder remains stable even when generating complex SVGs with many components.

Limitations and Future Work

Despite its success, the authors identified areas for improvement. Currently, the model lacks explicit semantic knowledge of components (e.g., it doesn't "know" a path is a "leaf"). Future work may involve adding semantic labels to the library and incorporating a specialized geometric loss function to further refine the mathematical accuracy of component placement.

Summary updated

BibTeX
@inproceedings{chen2025svgbuilder,
  author = {Zehao Chen and Rong Pan},
  title = {{SVGBuilder: Component-Based Colored SVG Generation with Text-Guided Autoregressive Transformers}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i3.32236},
}
Figure