VectorTalker: SVG Talking Face Generation with Progressive Vectorisation

Hao Hu, Xuan Wang, Jingxiang Sun, Yanbo Fan, Yu Guo, Caigui JiangarXiv 2023 · DOI · arXiv

Abstract

VectorTalker is the first framework to generate audio-driven talking heads directly in the Scalable Vector Graphics (SVG) domain. By combining a progressive coarse-to-fine vectorization strategy with a novel curve-splitting deformation technique, it produces resolution-independent animations that remain perfectly sharp at any scale. Unlike traditional raster-based methods, VectorTalker enables easy semantic editing and avoids common artifacts associated with pixel warping.

Introduction

Most state-of-the-art "talking head" models, such as SadTalker or MakeItTalk, operate in the raster domain, manipulating grids of pixels. This approach has inherent limitations: zooming leads to blurriness, and pixel warping often introduces distortions. Furthermore, raster images lack the semantic structure required for easy editing of individual facial features like eye shape or clothing color.

Vector graphics (SVG) address these issues by representing images as mathematical paths (Bezier curves). However, animating SVGs based on audio is challenging. It requires accurate vectorization of a portrait followed by natural-looking deformation of mathematical paths without breaking underlying geometry or creating visual overlaps.

Methodology

VectorTalker employs a two-stage pipeline to bridge the gap between static portraits and dynamic vector animations.

The VectorTalker pipeline consists of Progressive Vectorization and SVG Talking Animation stages.

Progressive Vectorization

To convert a standard photo into an SVG, VectorTalker uses a "coarse-to-fine" optimization. The process begins by fitting vector paths to a simplified, smoothed version of the image using $l_0$ regularization. As the optimization progresses, more paths are added to capture finer details. This prevents the model from getting stuck on local minima early on. Additionally, the Segment Anything Model (SAM) is used for semantic layering, ensuring the background remains stable while the face moves.

Landmark-Driven Animation

The system predicts facial landmark offsets from audio. To deform the SVG paths accordingly, it uses Delaunay Triangulation to create a mesh. A critical innovation is Curve Segmentation: since a single Bezier curve might span multiple triangles, the model mathematically splits these curves at triangle edges. This ensures each segment undergoes a consistent geometric transformation, preventing the "messy" visual artifacts seen in previous vector animation attempts.

Experiments and Results

The framework was evaluated on a benchmark of 20 portraits across diverse styles, including manga, cartoons, and photorealistic images.

VectorTalker demonstrates high-fidelity vectorization and animation across various artistic styles.

VectorTalker outperformed existing vectorization methods like DiffVG and LIVE. It achieved a Mean Squared Error (MSE) of 0.00131, significantly lower than DiffVG’s 0.00213, while maintaining higher PSNR and SSIM scores. Qualitatively, the animations remain sharp and clear compared to raster-based warping, which often blurs around the mouth and eyes during speech. Ablation studies confirmed that increasing smoothing levels (from 1 to 5) improved PSNR from 27.7 to 29.8, validating the progressive strategy.

Conclusion

VectorTalker introduces a robust method for generating resolution-independent talking heads. While the optimization-based vectorization is computationally intensive, the resulting SVGs offer unprecedented editability and clarity. Future work may involve incorporating 3D human priors to enhance complex expressions and handling highly dynamic elements like flowing hair or micro-expressions.

Summary updated

Abstract

High-fidelity and efficient audio-driven talking head generation has been a key research topic in computer graphics and computer vision. In this work, we study vector image based audio-driven talking head generation. Compared with directly animating the raster image that most widely used in existing works, vector image enjoys its excellent scalability being used for many applications. There are two main challenges for vector image based talking head generation: the high-quality vector image reconstruction w.r.t. the source portrait image and the vivid animation w.r.t. the audio signal. To address these, we propose a novel scalable vector graphic reconstruction and animation method, dubbed VectorTalker. Specifically, for the highfidelity reconstruction, VectorTalker hierarchically reconstructs the vector image in a coarse-to-fine manner. For the vivid audio-driven facial animation, we propose to use facial landmarks as intermediate motion representation and propose an efficient landmark-driven vector image deformation module. Our approach can handle various styles of portrait images within a unified framework, including Japanese manga, cartoon, and photorealistic images. We conduct extensive quantitative and qualitative evaluations and the experimental results demonstrate the superiority of VectorTalker in both vector graphic reconstruction and audio-driven animation.
BibTeX
@misc{hu2023vectortalker,
  author = {Hao Hu and Xuan Wang and Jingxiang Sun and Yanbo Fan and Yu Guo and Caigui Jiang},
  title = {{VectorTalker: SVG Talking Face Generation with Progressive Vectorisation}},
  year = {2023},
  eprint = {2312.11568},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.11568},
}
Figure