Abstract
VectorTalker is the first framework to generate audio-driven talking heads directly in the Scalable Vector Graphics (SVG) domain. By combining a progressive coarse-to-fine vectorization strategy with a novel curve-splitting deformation technique, it produces resolution-independent animations that remain perfectly sharp at any scale. Unlike traditional raster-based methods, VectorTalker enables easy semantic editing and avoids common artifacts associated with pixel warping.
Introduction
Most state-of-the-art "talking head" models, such as SadTalker or MakeItTalk, operate in the raster domain, manipulating grids of pixels. This approach has inherent limitations: zooming leads to blurriness, and pixel warping often introduces distortions. Furthermore, raster images lack the semantic structure required for easy editing of individual facial features like eye shape or clothing color.
Vector graphics (SVG) address these issues by representing images as mathematical paths (Bezier curves). However, animating SVGs based on audio is challenging. It requires accurate vectorization of a portrait followed by natural-looking deformation of mathematical paths without breaking underlying geometry or creating visual overlaps.
Methodology
VectorTalker employs a two-stage pipeline to bridge the gap between static portraits and dynamic vector animations.
Progressive Vectorization
To convert a standard photo into an SVG, VectorTalker uses a "coarse-to-fine" optimization. The process begins by fitting vector paths to a simplified, smoothed version of the image using $l_0$ regularization. As the optimization progresses, more paths are added to capture finer details. This prevents the model from getting stuck on local minima early on. Additionally, the Segment Anything Model (SAM) is used for semantic layering, ensuring the background remains stable while the face moves.
Landmark-Driven Animation
The system predicts facial landmark offsets from audio. To deform the SVG paths accordingly, it uses Delaunay Triangulation to create a mesh. A critical innovation is Curve Segmentation: since a single Bezier curve might span multiple triangles, the model mathematically splits these curves at triangle edges. This ensures each segment undergoes a consistent geometric transformation, preventing the "messy" visual artifacts seen in previous vector animation attempts.
Experiments and Results
The framework was evaluated on a benchmark of 20 portraits across diverse styles, including manga, cartoons, and photorealistic images.
VectorTalker outperformed existing vectorization methods like DiffVG and LIVE. It achieved a Mean Squared Error (MSE) of 0.00131, significantly lower than DiffVG’s 0.00213, while maintaining higher PSNR and SSIM scores. Qualitatively, the animations remain sharp and clear compared to raster-based warping, which often blurs around the mouth and eyes during speech. Ablation studies confirmed that increasing smoothing levels (from 1 to 5) improved PSNR from 27.7 to 29.8, validating the progressive strategy.
Conclusion
VectorTalker introduces a robust method for generating resolution-independent talking heads. While the optimization-based vectorization is computationally intensive, the resulting SVGs offer unprecedented editability and clarity. Future work may involve incorporating 3D human priors to enhance complex expressions and handling highly dynamic elements like flowing hair or micro-expressions.
Summary updated