Towards High-fidelity 3D Talking Avatar with Personalized Dynamic Texture

Xuanchen Li, Jianyu Wang, Yuhao Cheng, Yikun Zeng, Xingyu Ren, Wenhan Zhu, Weiming Zhao, Yichao Yan

TL;DR

While speech-driven 3D talking heads have come a long way, most methods only animate the underlying 3D mesh, leaving facial textures static and creating an uncanny valley effect. This paper introduces TexTalk4D, a high-resolution 4D dataset featuring 8K dynamic textures, alongside TexTalker, a diffusion-based framework that simultaneously synthesizes realistic facial geometry and synchronized, personalized wrinkle dynamics directly from audio.

What Problem Does This Paper Address?

When we talk, our faces don't just move structurally; our skin stretches, compresses, and forms micro-wrinkles around the mouth, eyes, and nose. Traditional speech-driven 3D facial animation models focus almost exclusively on predicting 3D vertex coordinates or expression coefficients. They apply these movements to a static texture map, which strips away the subtle, realistic skin behaviors that signal natural human speech.

The primary bottlenecks preventing progress in this area have been twofold:

  1. Data Scarcity: Capturing high-fidelity, temporally consistent 3D geometry paired with ultra-high-resolution dynamic textures is incredibly difficult and expensive. Existing datasets either lack texture resolution or lack diversity.
  2. Cross-Modality Co-generation: Simultaneously generating two vastly different modalities—discrete 3D mesh structures and continuous, high-dimensional texture maps—while keeping them perfectly synchronized with audio is a massive algorithmic challenge.

Methods

TexTalker solves the co-generation problem by mapping both geometry and texture into a unified, low-dimensional latent space before feeding them into a diffusion model.

Figure 3. The overview of TexTalker. (a) We train quantized autoencoders to unify the representation of geometry and texture with better efficiency. (b) Based on the learned low-dimensional animation primitives, we employ an LDM to jointly diffuse geometry and texture latent offsets ∆z from the style pivots p for long-term correlation learning. (c) By adding back the style pivots, the motion and wrinkle styles can be independently controlled. Finally, the personalized textured animation assets can be obtained by decoders.

The framework operates in three main stages:

  1. Facial Animation Primitive Learning: 3D vertex offsets are projected into 2D UV space as "motion maps," while texture variations are represented as "wrinkle maps" (isolating speech-driven changes from identity bias). Two independent VQGAN-style quantized autoencoders compress these maps into discrete latent spaces (codebooks), capturing the essential primitives of facial animation.
  2. Motion-Wrinkle Latent Diffusion Model (LDM): The motion and wrinkle latents are concatenated and processed by a transformer-based LDM. The model is conditioned on speech features extracted via a pre-trained HuBERT audio encoder. An alignment mask ensures tight frame-by-frame audio-visual synchronization, while a windowed diffusion approach maintains long-term temporal consistency.
  3. Pivot-Based Disentangled Style Injection: To control the style of the avatar independently from the speech content, the authors introduce "style pivots." By averaging a sequence of latent codes for a specific identity, they establish a centroid that represents that person's unique speaking and wrinkling style. The LDM learns to predict style-free offsets from this pivot, allowing users to mix and match the speaking style of one person with the wrinkling style of another.

Datasets

To train TexTalker, the authors captured and released a brand-new benchmark:

  • TexTalk4D: A high-precision, audio-mesh-texture-aligned dataset featuring 100 minutes of data from 100 subjects (60 males, 40 females). Each subject recorded one minute of Mandarin speech at 60 FPS. The dataset provides scan-level whole-head meshes (8,280 vertices) paired with clean, 8K-resolution dynamic textures capturing pore-level details.

Data was captured using a 24-camera LightStage system and processed using state-of-the-art dynamic face reconstruction (Topo4D) to ensure perfect topological and temporal consistency.

Figure 2. Data processing pipeline. We employ Topo4D [38] to obtain consistent meshes from LightStage captures. Based on the mesh sequence, we map the color from 24-view 4K images to get the 8K textures. Mesh alignment and texture blending are then conducted to get the final assets.

Key Insights and Ideas

  • The Synergy of Joint Learning: A key finding from the ablation studies is that generating motion and texture together actually improves the accuracy of both. Geometry and texture are inherently correlated; by modeling them jointly, the network resolves cross-modal ambiguities that cause separate models to degrade.
  • Independent Latent Spaces are Better: While joint generation is beneficial, forcing motion and texture into the same latent codebook hurts performance. Keeping their primitive spaces independent but diffusing them jointly yields the best expressiveness.
  • Style is a Centroid: The authors discovered that identity-specific styles naturally cluster within the learned VQ latent space. This means a user doesn't need complex style-extraction networks or fine-tuning; simply averaging a few seconds of a subject's latent codes creates a highly effective "style pivot."

Key Results

TexTalker was evaluated against state-of-the-art geometry-only methods (FaceFormer, CodeTalker, FaceDiffuser) and image-translation texture baselines.

Geometry Quality

On the TexTalk4D test set, TexTalker achieved top-tier performance across standard metrics (lower is better):

  • Lip Vertex Error (LVE): 1.49 $\times 10^{-2}$mm (outperforming FaceDiffuser's 1.53 and CodeTalker's 1.83).
  • Mean Vertex Error (MVE): 2.34 $\times 10^{-2}$mm (outperforming FaceDiffuser's 2.38).
  • Upper-face Dynamics Deviation (FDD): 1.20 $\times 10^{-3}$mm, demonstrating significantly more realistic upper-face and eye-blinking dynamics.

Texture Quality

When compared to generating textures via image-to-image translation (Pix2PixHD), TexTalker demonstrated superior clarity and realism:

  • PSNR: 44.13 dB vs. 42.34 dB (baseline).
  • LPIPS (Perceptual Loss): 0.0101 vs. 0.0187 (baseline).

Qualitatively, user studies confirmed that TexTalker produces the most realistic animations with the highest geometry-texture consistency.

Limitations and Future Work

Despite the impressive high-fidelity results, the authors note a few limitations:

  • Demographic and Language Bias: The TexTalk4D dataset currently consists primarily of Asian youth speaking Mandarin. While the model shows decent zero-shot generalization to other races and languages, expanding the dataset's diversity is a necessary next step.
  • Inference Speed: While the LDM generates latent features at a real-time capable 24 FPS, the final texture generation and rendering pipeline runs at 38 FPS on an NVIDIA 3090. Further optimization will be required for deployment in interactive, real-time applications like video games or virtual assistants.

Summary updated

Abstract

Significant progress has been made for speech-driven 3D face animation, but most works focus on learning the motion of mesh/geometry, ignoring the impact of dynamic texture. In this work, we reveal that dynamic texture plays a key role in rendering high-fidelity talking avatars, and introduce a high-resolution 4D dataset \textbf{TexTalk4D}, consisting of 100 minutes of audio-synced scan-level meshes with detailed 8K dynamic textures from 100 subjects. Based on the dataset, we explore the inherent correlation between motion and texture, and propose a diffusion-based framework \textbf{TexTalker} to simultaneously generate facial motions and dynamic textures from speech. Furthermore, we propose a novel pivot-based style injection strategy to capture the complicity of different texture and motion styles, which allows disentangled control. TexTalker, as the first method to generate audio-synced facial motion with dynamic texture, not only outperforms the prior arts in synthesising facial motions, but also produces realistic textures that are consistent with the underlying facial movements. Project page: https://xuanchenli.github.io/TexTalk/.
BibTeX
@inproceedings{li2025towards,
  author = {Xuanchen Li and Jianyu Wang and Yuhao Cheng and Yikun Zeng and Xingyu Ren and Wenhan Zhu and Weiming Zhao and Yichao Yan},
  title = {{Towards High-fidelity 3D Talking Avatar with Personalized Dynamic Texture}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00028},
}
Figure