TL;DR
This paper introduces Style²Talker, a novel system for generating high-resolution talking head videos that simultaneously incorporate user-defined emotion and art styles. It addresses the limitations of existing methods by enabling control over both expressive nuances and artistic aesthetics, driven by simple text and image inputs.
What Problem Does This Paper Address?
Creating realistic and expressive talking head videos is a challenging task with applications ranging from virtual avatars to film production. While previous research has focused on accurately synchronizing lip movements with audio, it has largely overlooked the crucial elements of emotion and art style. Existing methods either offer limited control over emotions (e.g., one-hot labels) or require additional, often inconvenient, video inputs for emotional expression. Similarly, while art style transfer has been explored for static images, its application to continuous video generation with synchronized speech and emotion remains underdeveloped. This paper aims to bridge this gap by providing a unified framework that allows users to control both the emotional expressiveness and the artistic rendering of talking head videos, using intuitive text and image prompts.
Methods
Style²Talker employs a two-stage approach: Style-E for emotion style transfer and Style-A for art style transfer.
- Style-E (Emotion Style Transfer): This stage utilizes a latent diffusion model to generate 3D Morphable Model (3DMM) expression coefficients. The diffusion model is conditioned on three inputs: a text description of the desired emotion, an identity image, and an audio clip. A key innovation here is a labor-free paradigm for generating detailed emotion text descriptions for existing datasets, leveraging large-scale pretrained models. This allows for more nuanced control over facial expressions than traditional methods. The diffusion model is designed for efficient inference by predicting motion coefficients directly rather than noise, and uses an MLP-based denoising network for speed.
- Style-A (Art Style Transfer): This stage builds upon the StyleGAN architecture, specifically a modified version called DualStyleGAN. It takes the generated emotion-driven 3DMM coefficients and an art style source picture as input. To ensure high-resolution and consistent artistic style, the system incorporates:
- A coefficient-driven motion generator that produces flow fields to warp StyleGAN's intermediate feature maps, aligning them with the desired motion.
- A content encoder to extract multi-level features from the identity image, which are then fed into StyleGAN via skip connections to preserve fine details.
- A refinement network to mitigate potential artifacts arising from feature map warping.
Datasets
- MEAD: Used for training the Style-E stage. The authors note that while MEAD provides audio-visual data, they augment it with automatically generated text descriptions for emotion styles.
- HDTF: Used for training the Style-A stage.
- Art Datasets: Various art datasets were used to obtain art style references.
The paper highlights the creation of a "labor-free paradigm" for automatically annotating emotional text labels for existing audio-visual datasets, which is a significant contribution to enabling text-driven emotion control.
Key Insights and Ideas
- Unified Control for Emotion and Art Style: The core insight is that combining emotion and art style control in a single talking head generation system is feasible and highly desirable for richer, more personalized outputs.
- Text-Driven Emotion Generation: Using text descriptions for emotion style offers a more flexible and user-friendly alternative to one-hot labels or emotion videos.
- Diffusion Models for Expressive Motion: Latent diffusion models prove effective for generating nuanced and controllable 3DMM coefficients that capture desired emotional expressions.
- StyleGAN Adaptation for Video: Modifying StyleGAN with motion-driven feature warping and content preservation mechanisms is crucial for generating high-resolution, artistically stylized talking head videos.
- Labor-Free Data Annotation: The development of an automated pipeline for generating emotion text descriptions significantly lowers the barrier to entry for text-controlled emotion synthesis.
Key Results
- Quantitative Performance: Style²Talker outperforms several state-of-the-art methods in terms of SSIM, FID, and M-LMD (lip synchronization) on both MEAD and HDTF datasets. It achieves a suboptimal Sync.conf score, ranking second to methods specifically optimized for lip sync.
- Qualitative Performance: Visual comparisons demonstrate that Style²Talker generates high-resolution talking head videos with accurate lip synchronization, desired emotion styles, and compelling art style transfer, often surpassing existing methods in visual fidelity and coherence.
- User Study: User studies indicated a strong preference for Style²Talker in terms of art style and lip synchronization, and competitive performance in emotion style matching compared to methods using more informative emotion video inputs.
- Ablation Studies: Experiments confirmed the importance of each component: excluding emotion style led to less expressive results, removing the art path resulted in unchanged art style, replacing the diffusion model with a GAN degraded lip sync, and removing skip connections or the refinement network reduced image quality and visual coherence.
Limitations and Future Work
- Inference Time: While the diffusion model is optimized for faster inference, it can still be computationally intensive compared to GAN-based methods. The paper mentions using DDIM sampling to reduce inference steps.
- Data Dependency: The quality of the generated emotion style is dependent on the accuracy and expressiveness of the automatically generated text descriptions.
- Future Work: The authors suggest that their work inspires further research into leveraging large-scale pretrained models for more practical and captivating explorations in talking head generation. The system's ability to combine multiple styles opens avenues for exploring even more complex and nuanced stylistic controls.
Summary updated