TL;DR
This paper introduces EmoStyle, a novel method for one-shot facial expression editing that offers fine-grained control over emotions using continuous valence and arousal parameters. EmoStyle disentangles emotion from other facial attributes and can edit real-world images, producing high-resolution, realistic results while preserving identity and background.
What Problem Does This Paper Address?
Editing facial expressions in images has seen significant progress, but existing methods often struggle with providing nuanced control over emotions. Many approaches generate only a limited set of discrete expressions or lack the ability to precisely control the intensity and type of emotion. Furthermore, editing real-world images, especially those not generated by the same model, presents a challenge due to domain differences. Researchers and practitioners in fields like psychology and visual effects require tools that can synthesize realistic, diverse, and scientifically grounded emotional expressions with high fidelity and identity preservation.
Methods
EmoStyle employs a two-phase approach:
Phase 1: Training EmoExtract This phase focuses on training an "EmoExtract" module. The core idea is to disentangle emotional expression from other facial characteristics. This is achieved by training a module that takes a latent code representing a face (from StyleGAN2's W space) and modifies it based on continuous valence and arousal parameters. The training alternates between:
- Emotion Variation: Generating faces with random valence-arousal values to encourage diversity and learn to synthesize emotions not frequently seen in the dataset. This phase incorporates a novel background loss to preserve background, hairstyle, and skin color.
- Emotion Reconstruction: Reconstructing the input face using its original emotion parameters to ensure identity and overall image fidelity.
Phase 2: Fine-tuning StyleGAN2 for Real Images To enable editing of real-world images (out-of-domain for the pre-trained StyleGAN2), EmoStyle uses an adapted inversion method. The latent code of the real image is first extracted. Then, the pre-trained StyleGAN2 generator is fine-tuned on this single image, guided by the EmoExtract module and the desired emotion parameters. This fine-tuning allows the generator to adapt to the specific face while learning to manipulate its latent representation for expression editing.
Datasets
- Generated Images: The paper utilizes 70,000 generated images and their latent codes for training, and a separate set of 1,000 images for testing. These images are generated using StyleGAN2, originally trained on the FFHQ dataset.
- Real Images: For qualitative real-image experiments, a subset of the CelebA dataset is used.
Key Insights and Ideas
- Continuous Emotion Control: Moving beyond discrete emotions to valence and arousal provides a richer, more nuanced way to specify and synthesize emotional expressions.
- Disentanglement is Key: Separating emotional expression from other facial attributes (like identity, pose, and background) is crucial for realistic editing.
- Background Loss for Realism: A novel background loss function is essential for maintaining consistency in non-facial elements, preventing unwanted artifacts during extreme emotion edits.
- One-Shot Editing of Real Faces: Adapting StyleGAN2's latent space through fine-tuning on a single real image is an effective strategy for editing faces outside the original training domain.
- StyleGAN2's Rich Latent Space: Leveraging the expressive latent space of StyleGAN2 is fundamental to achieving high-quality and controllable edits.
Key Results
- Quantitative Performance: EmoStyle outperforms existing methods (GANmut, StyleCLIP, L2L, GANSpace, InterFaceGAN, StyleFlow) in terms of FID (Fréchet Inception Distance), LPIPS (Learned Perceptual Image Patch Similarity), and identity preservation (measured by ArcFace). Specifically, EmoStyle achieves an FID of 7.86 and an LPIPS of 0.07.
- Emotional Diversity: EmoStyle demonstrates greater diversity in generated facial expressions, achieving standard deviations of 0.5 for valence and 0.25 for arousal, compared to StyleGAN2's 0.42 and 0.15 respectively.
- High-Resolution Output: The method generates high-quality images at a resolution of 1024 x 1024 pixels.
- Qualitative Results: Visual comparisons show EmoStyle's ability to synthesize a wide range of expressions, effectively conveying valence and arousal, while preserving identity and background, even for challenging edits.
Limitations and Future Work
- Dominance Dimension: The paper notes that the current valence-arousal model does not incorporate the "dominance" dimension of emotions, suggesting it as a promising avenue for future research.
- Action Units (AUs): The authors propose exploring Action Units (AUs) as an additional control axis, allowing for more localized and fine-grained control over facial expressions beyond global semantic editing.
- Identity Preservation in Extreme Edits: While generally good, the paper acknowledges that in some extreme emotion edits, identity preservation can still be a challenge, especially when certain modules are ablated.
- Inversion Method Dependence: The performance on real images is partly dependent on the chosen inversion method (e.g., pSp), and further exploration of robust inversion techniques could be beneficial.
Summary updated