Abstract
Scalable Vector Graphics (SVG) present unique challenges for deep learning due to their continuous coordinate space and sequential command structure. Traditional transformer models often rely on discretization, leading to precision loss, and fail to capture non-sequential geometric relationships. We present SVGformer, a transformer-based architecture that operates directly on continuous SVG data. By integrating a 1D convolutional embedding with a geometric self-attention mechanism guided by the Medial Axis Transform (MAT), SVGformer effectively captures both local precision and global structural dependencies. Our model sets new benchmarks in SVG reconstruction, classification, and retrieval.
Introduction
While deep learning has achieved remarkable success in grid-based bitmap image processing, Scalable Vector Graphics (SVG) remain a challenging domain. SVGs are defined by mathematical paths—sequences of commands like "move to," "line to," and "cubic Bézier curve"—parameterized by continuous coordinates. Existing models typically suffer from two main issues: quantization errors from discretizing coordinates and "geometric blindness," where standard transformers fail to recognize physical adjacency between commands that are distant in the sequence.
SVGformer addresses these gaps by treating SVGs as continuous sequences and injecting structural inductive biases to capture the underlying geometry of vector shapes.
Methodology
SVGformer introduces several key innovations to handle the complexities of vector data:
- Continuous Feature Embedding: Instead of converting coordinates into discrete tokens, SVGformer employs a 1D Convolutional layer to map raw, continuous SVG parameters into a high-dimensional embedding space, preserving the original data's precision.
- Geometric Information Extraction: The model utilizes the Medial Axis Transform (MAT) to identify the "skeleton" of a shape. This process labels SVG segments to indicate which commands belong to the same geometric structure.
- Geometric Self-Attention: This novel mechanism incorporates an adjacency matrix derived from MAT labels. A Graph Convolutional Network (GCN) calculates relationship scores between commands, which are then injected into the transformer's attention mechanism. This forces the model to attend to geometrically related commands, regardless of their sequence distance.
- Sparse Query Attention: To manage long sequences efficiently, the architecture adopts a sparse attention mechanism, reducing the quadratic memory overhead associated with standard transformers.
Experiments and Results
The model was evaluated on the Google Fonts dataset (62 glyph categories across 378 font families) and a diverse Icons dataset (59 categories). SVGformer demonstrated significant improvements over existing state-of-the-art models:
- Reconstruction: Accuracy improved by 17.8% (measured by Chamfer Distance) compared to baselines.
- Classification: In identifying font families, SVGformer outperformed previous models by 51.2%.
- Retrieval: The model showed a 42.5% improvement in unsupervised style retrieval tasks over DeepSVG.
Qualitatively, SVGformer excels at reconstructing fine details, such as sharp corners and smooth curves, which are often lost in discretized models. It also enables smoother transitions during font interpolation.
Conclusion
SVGformer demonstrates that incorporating geometric structural information is vital for learning effective representations of complex vector shapes. By maintaining continuous values and utilizing geometric self-attention, the model achieves superior performance in both content and style-related tasks. Future work may involve integrating diffusion models to further enhance the smoothness of the latent space for advanced vector generation tasks.
Paper Citation: Defu Cao, Zhaowen Wang, Jose Echevarria, and Yan Liu. "SVGformer: Representation Learning for Continuous Vector Graphics using Transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Summary updated