Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style

Shuai Tan, Bin Ji, Ye Pan

TL;DR

This paper introduces Style²Talker, a novel system for generating high-resolution talking head videos that simultaneously incorporate user-defined emotion and art styles. It addresses the limitations of existing methods by enabling control over both expressive nuances and artistic aesthetics, driven by simple text and image inputs.

What Problem Does This Paper Address?

Creating realistic and expressive talking head videos is a challenging task with applications ranging from virtual avatars to film production. While previous research has focused on accurately synchronizing lip movements with audio, it has largely overlooked the crucial elements of emotion and art style. Existing methods either offer limited control over emotions (e.g., one-hot labels) or require additional, often inconvenient, video inputs for emotional expression. Similarly, while art style transfer has been explored for static images, its application to continuous video generation with synchronized speech and emotion remains underdeveloped. This paper aims to bridge this gap by providing a unified framework that allows users to control both the emotional expressiveness and the artistic rendering of talking head videos, using intuitive text and image prompts.

Methods

Style²Talker employs a two-stage approach: Style-E for emotion style transfer and Style-A for art style transfer.

  • Style-E (Emotion Style Transfer): This stage utilizes a latent diffusion model to generate 3D Morphable Model (3DMM) expression coefficients. The diffusion model is conditioned on three inputs: a text description of the desired emotion, an identity image, and an audio clip. A key innovation here is a labor-free paradigm for generating detailed emotion text descriptions for existing datasets, leveraging large-scale pretrained models. This allows for more nuanced control over facial expressions than traditional methods. The diffusion model is designed for efficient inference by predicting motion coefficients directly rather than noise, and uses an MLP-based denoising network for speed.
  • Style-A (Art Style Transfer): This stage builds upon the StyleGAN architecture, specifically a modified version called DualStyleGAN. It takes the generated emotion-driven 3DMM coefficients and an art style source picture as input. To ensure high-resolution and consistent artistic style, the system incorporates:
  • A coefficient-driven motion generator that produces flow fields to warp StyleGAN's intermediate feature maps, aligning them with the desired motion.
  • A content encoder to extract multi-level features from the identity image, which are then fed into StyleGAN via skip connections to preserve fine details.
  • A refinement network to mitigate potential artifacts arising from feature map warping.

Datasets

  • MEAD: Used for training the Style-E stage. The authors note that while MEAD provides audio-visual data, they augment it with automatically generated text descriptions for emotion styles.
  • HDTF: Used for training the Style-A stage.
  • Art Datasets: Various art datasets were used to obtain art style references.

The paper highlights the creation of a "labor-free paradigm" for automatically annotating emotional text labels for existing audio-visual datasets, which is a significant contribution to enabling text-driven emotion control.

Key Insights and Ideas

  • Unified Control for Emotion and Art Style: The core insight is that combining emotion and art style control in a single talking head generation system is feasible and highly desirable for richer, more personalized outputs.
  • Text-Driven Emotion Generation: Using text descriptions for emotion style offers a more flexible and user-friendly alternative to one-hot labels or emotion videos.
  • Diffusion Models for Expressive Motion: Latent diffusion models prove effective for generating nuanced and controllable 3DMM coefficients that capture desired emotional expressions.
  • StyleGAN Adaptation for Video: Modifying StyleGAN with motion-driven feature warping and content preservation mechanisms is crucial for generating high-resolution, artistically stylized talking head videos.
  • Labor-Free Data Annotation: The development of an automated pipeline for generating emotion text descriptions significantly lowers the barrier to entry for text-controlled emotion synthesis.

Key Results

  • Quantitative Performance: Style²Talker outperforms several state-of-the-art methods in terms of SSIM, FID, and M-LMD (lip synchronization) on both MEAD and HDTF datasets. It achieves a suboptimal Sync.conf score, ranking second to methods specifically optimized for lip sync.
  • Qualitative Performance: Visual comparisons demonstrate that Style²Talker generates high-resolution talking head videos with accurate lip synchronization, desired emotion styles, and compelling art style transfer, often surpassing existing methods in visual fidelity and coherence.
  • User Study: User studies indicated a strong preference for Style²Talker in terms of art style and lip synchronization, and competitive performance in emotion style matching compared to methods using more informative emotion video inputs.
  • Ablation Studies: Experiments confirmed the importance of each component: excluding emotion style led to less expressive results, removing the art path resulted in unchanged art style, replacing the diffusion model with a GAN degraded lip sync, and removing skip connections or the refinement network reduced image quality and visual coherence.

Limitations and Future Work

  • Inference Time: While the diffusion model is optimized for faster inference, it can still be computationally intensive compared to GAN-based methods. The paper mentions using DDIM sampling to reduce inference steps.
  • Data Dependency: The quality of the generated emotion style is dependent on the accuracy and expressiveness of the automatically generated text descriptions.
  • Future Work: The authors suggest that their work inspires further research into leveraging large-scale pretrained models for more practical and captivating explorations in talking head generation. The system's ability to combine multiple styles opens avenues for exploring even more complex and nuanced stylistic controls.

Summary updated

Abstract

Although automatically animating audio-driven talking heads has recently received growing interest, previous efforts have mainly concentrated on achieving lip synchronization with the audio, neglecting two crucial elements for generating expressive videos: emotion style and art style. In this paper, we present an innovative audio-driven talking face generation method called Style2Talker. It involves two stylized stages, namely Style-E and Style-A, which integrate text-controlled emotion style and picture-controlled art style into the final output. In order to prepare the scarce emotional text descriptions corresponding to the videos, we propose a labor-free paradigm that employs large-scale pretrained models to automatically annotate emotional text labels for existing audiovisual datasets. Incorporating the synthetic emotion texts, the Style-E stage utilizes a large-scale CLIP model to extract emotion representations, which are combined with the audio, serving as the condition for an efficient latent diffusion model designed to produce emotional motion coefficients of a 3DMM model. Moving on to the Style-A stage, we develop a coefficient-driven motion generator and an art-specific style path embedded in the well-known StyleGAN. This allows us to synthesize high-resolution artistically stylized talking head videos using the generated emotional motion coefficients and an art style source picture. Moreover, to better preserve image details and avoid artifacts, we provide StyleGAN with the multi-scale content features extracted from the identity image and refine its intermediate feature maps by the designed content encoder and refinement network, respectively. Extensive experimental results demonstrate our method outperforms existing state-of-the-art methods in terms of audio-lip synchronization and performance of both emotion style and art style.
BibTeX
@inproceedings{tan2024style,
  author = {Shuai Tan and Bin Ji and Ye Pan},
  title = {{Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=0TlDUKqjQ3},
}
Figure