TL;DR
While speech-driven 3D talking heads have come a long way, most methods only animate the underlying 3D mesh, leaving facial textures static and creating an uncanny valley effect. This paper introduces TexTalk4D, a high-resolution 4D dataset featuring 8K dynamic textures, alongside TexTalker, a diffusion-based framework that simultaneously synthesizes realistic facial geometry and synchronized, personalized wrinkle dynamics directly from audio.
What Problem Does This Paper Address?
When we talk, our faces don't just move structurally; our skin stretches, compresses, and forms micro-wrinkles around the mouth, eyes, and nose. Traditional speech-driven 3D facial animation models focus almost exclusively on predicting 3D vertex coordinates or expression coefficients. They apply these movements to a static texture map, which strips away the subtle, realistic skin behaviors that signal natural human speech.
The primary bottlenecks preventing progress in this area have been twofold:
- Data Scarcity: Capturing high-fidelity, temporally consistent 3D geometry paired with ultra-high-resolution dynamic textures is incredibly difficult and expensive. Existing datasets either lack texture resolution or lack diversity.
- Cross-Modality Co-generation: Simultaneously generating two vastly different modalities—discrete 3D mesh structures and continuous, high-dimensional texture maps—while keeping them perfectly synchronized with audio is a massive algorithmic challenge.
Methods
TexTalker solves the co-generation problem by mapping both geometry and texture into a unified, low-dimensional latent space before feeding them into a diffusion model.
The framework operates in three main stages:
- Facial Animation Primitive Learning: 3D vertex offsets are projected into 2D UV space as "motion maps," while texture variations are represented as "wrinkle maps" (isolating speech-driven changes from identity bias). Two independent VQGAN-style quantized autoencoders compress these maps into discrete latent spaces (codebooks), capturing the essential primitives of facial animation.
- Motion-Wrinkle Latent Diffusion Model (LDM): The motion and wrinkle latents are concatenated and processed by a transformer-based LDM. The model is conditioned on speech features extracted via a pre-trained HuBERT audio encoder. An alignment mask ensures tight frame-by-frame audio-visual synchronization, while a windowed diffusion approach maintains long-term temporal consistency.
- Pivot-Based Disentangled Style Injection: To control the style of the avatar independently from the speech content, the authors introduce "style pivots." By averaging a sequence of latent codes for a specific identity, they establish a centroid that represents that person's unique speaking and wrinkling style. The LDM learns to predict style-free offsets from this pivot, allowing users to mix and match the speaking style of one person with the wrinkling style of another.
Datasets
To train TexTalker, the authors captured and released a brand-new benchmark:
- TexTalk4D: A high-precision, audio-mesh-texture-aligned dataset featuring 100 minutes of data from 100 subjects (60 males, 40 females). Each subject recorded one minute of Mandarin speech at 60 FPS. The dataset provides scan-level whole-head meshes (8,280 vertices) paired with clean, 8K-resolution dynamic textures capturing pore-level details.
Data was captured using a 24-camera LightStage system and processed using state-of-the-art dynamic face reconstruction (Topo4D) to ensure perfect topological and temporal consistency.
Key Insights and Ideas
- The Synergy of Joint Learning: A key finding from the ablation studies is that generating motion and texture together actually improves the accuracy of both. Geometry and texture are inherently correlated; by modeling them jointly, the network resolves cross-modal ambiguities that cause separate models to degrade.
- Independent Latent Spaces are Better: While joint generation is beneficial, forcing motion and texture into the same latent codebook hurts performance. Keeping their primitive spaces independent but diffusing them jointly yields the best expressiveness.
- Style is a Centroid: The authors discovered that identity-specific styles naturally cluster within the learned VQ latent space. This means a user doesn't need complex style-extraction networks or fine-tuning; simply averaging a few seconds of a subject's latent codes creates a highly effective "style pivot."
Key Results
TexTalker was evaluated against state-of-the-art geometry-only methods (FaceFormer, CodeTalker, FaceDiffuser) and image-translation texture baselines.
Geometry Quality
On the TexTalk4D test set, TexTalker achieved top-tier performance across standard metrics (lower is better):
- Lip Vertex Error (LVE): 1.49 $\times 10^{-2}$mm (outperforming FaceDiffuser's 1.53 and CodeTalker's 1.83).
- Mean Vertex Error (MVE): 2.34 $\times 10^{-2}$mm (outperforming FaceDiffuser's 2.38).
- Upper-face Dynamics Deviation (FDD): 1.20 $\times 10^{-3}$mm, demonstrating significantly more realistic upper-face and eye-blinking dynamics.
Texture Quality
When compared to generating textures via image-to-image translation (Pix2PixHD), TexTalker demonstrated superior clarity and realism:
- PSNR: 44.13 dB vs. 42.34 dB (baseline).
- LPIPS (Perceptual Loss): 0.0101 vs. 0.0187 (baseline).
Qualitatively, user studies confirmed that TexTalker produces the most realistic animations with the highest geometry-texture consistency.
Limitations and Future Work
Despite the impressive high-fidelity results, the authors note a few limitations:
- Demographic and Language Bias: The TexTalk4D dataset currently consists primarily of Asian youth speaking Mandarin. While the model shows decent zero-shot generalization to other races and languages, expanding the dataset's diversity is a necessary next step.
- Inference Speed: While the LDM generates latent features at a real-time capable 24 FPS, the final texture generation and rendering pipeline runs at 38 FPS on an NVIDIA 3090. Further optimization will be required for deployment in interactive, real-time applications like video games or virtual assistants.
Summary updated