Abstract
FG-EmoTalk is a novel framework for one-shot talking head generation that provides fine-grained control over facial expressions. By leveraging Facial Action Units (AUs), the system allows users to manipulate specific muscle groups—such as raising an inner brow or pulling lip corners—to create nuanced, realistic, and subject-agnostic facial animations from a single source image. This approach moves beyond simple categorical emotion labels to capture the continuous physical state of human expression.
Introduction
While AI-generated talking heads have become remarkably realistic in terms of lip-syncing and image quality, controlling their emotions remains a challenge. Most existing methods rely on discrete labels (e.g., "happy" or "sad") or simply copy expressions from a driving video. These approaches fail to capture the complexity of human expression and often introduce unwanted artifacts from the driving actor's identity. FG-EmoTalk addresses these issues by providing a way to edit expressions at a granular level using the intensities of individual muscle movements.
Methodology
The core of FG-EmoTalk is a novel architecture designed for feature disentanglement and AU-based control.
Three-Way Disentanglement
Unlike standard methods that only separate "pose" and "expression," FG-EmoTalk isolates three distinct components: Appearance (identity/face shape), Pose (head orientation/shoulder movement), and Expression. This ensures that changing an expression doesn't accidentally change the person's identity or head tilt.
AU Encoder Module
The researchers use Facial Action Units (AUs) as the fundamental units of expression. An AU encoder takes an intensity vector (representing how much a specific muscle group is contracted) and transforms it into a latent code. This code can then be injected into the generation pipeline to modify specific facial regions.
Latent Code Editing and Training
The system fuses the disentangled pose and expression features with the appearance features in a shared latent space using an orthogonal dictionary. Since there isn't a massive dataset of "Person A making Expression B with Pose C," the authors designed a self-supervised strategy. They apply specific losses—like an Appearance Loss (using a ReID backbone) and an Expression Loss (using Micron-BERT)—to ensure each component is disentangled correctly.
Datasets and Evaluation
The model was trained and evaluated using high-profile datasets including HDTF, CelebV-HQ, DISFA (for AU intensity), and MEAD (for cross-dataset validation).
FG-EmoTalk outperforms state-of-the-art (SOTA) models like EAMM and VideoRetalking across multiple metrics:
- Facial Expression Quality: On the MEAD dataset, FG-EmoTalk achieved an emotion classification accuracy ($ACC_{emo}$) of 0.6102, significantly higher than EAMM (0.5435).
- Visual Realism: It achieved the highest scores in PSNR (31.47) and SSIM (0.7970), indicating better image reconstruction.
- User Preference: In a user study, FG-EmoTalk was selected as the "best" in terms of naturalness and expression over 60% of the time.
Conclusion
FG-EmoTalk demonstrates that targeting AUs allows for a level of "micro-expression" control that categorical models cannot reach. While challenges remain in data scarcity for AU annotations and consistent dental structure generation, this work provides a robust foundation for subject-agnostic, fine-grained facial animation.
Summary updated