MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learning

Yue Han, Junwei Zhu, Yuxiang Feng, Xiaozhong Ji, Keke He, Xiangtai Li, Zhucun Xue, Yong LiuarXiv 2024 · DOI · arXiv

Abstract

Generating realistic talking head videos remains a challenge due to the difficulty of maintaining identity preservation and temporal consistency, especially when training on public datasets. We present MIMAFace, a novel diffusion-based framework that overcomes these hurdles. By modulating appearance features with motion and identity signals and introducing an inter-clip affinity module, MIMAFace generates high-fidelity animations with superior stability. Our approach effectively bridges the gap between low-level generative features and high-level identity markers, outperforming current state-of-the-art methods in motion accuracy and identity preservation.

Introduction

Recent advancements in diffusion models have pushed the boundaries of face animation. Most existing methods utilize a "ReferenceNet" (a duplicated U-Net) to maintain the appearance of a source image. However, these models often struggle with two primary issues: the Data Gap and Temporal Discontinuity.

When trained on public datasets like VoxCeleb, current models frequently produce stiff expressions or lose the subject's identity. Furthermore, generating long-form video by stitching short clips often results in jarring "jumps" in lighting or semantics. MIMAFace addresses these by arguing that models must capture motion-related textures (e.g., facial wrinkles) and high-level identity markers more effectively than standard CLIP-based appearance encoders allow.

Methodology

MIMAFace introduces two core innovations to the standard diffusion pipeline: Motion-Identity Modulated Appearance Learning (MIA) and Inter-clip Affinity Learning (ICA).

Motion-Identity Modulated Appearance Learning (MIA)

Rather than relying solely on raw CLIP features, the MIA module modulates them using dynamic signals:

  • Motion Modulation: By incorporating 3D Morphable Model (3DMM) coefficients and gaze embeddings, the model learns how textures evolve with movement, such as skin folding during a smile.
  • Identity Modulation: To ensure the subject remains recognizable throughout the denoising process, we introduce an Identity Contrastive Loss ($L_{id}$). This forces the encoder to prioritize discriminative features that are invariant to lighting or pose changes.

Inter-clip Affinity Learning (ICA)

To eliminate flickering between video segments, the ICA module models the relationship between the current clip and the preceding one. By conditioning the model on the latent representations of previous frames during training, we ensure that new clips begin as a seamless continuation of the last, maintaining temporal flow across long sequences.

Overview of the MIMAFace framework, featuring the MIA module for appearance modulation and the ICA module for temporal consistency.

Experiments and Datasets

The model was trained on VoxCeleb2 and evaluated on VoxCeleb1. To test generalization and robustness, we utilized FFHQ for unseen identities, MPIE for extreme head poses, and VFHQ for high-definition (768x768) synthesis.

Key Results

Quantitative Performance

MIMAFace demonstrates state-of-the-art performance on the VoxCeleb1 benchmark, outperforming popular methods like AniPortrait and FADM. Specifically, it achieved the lowest error rates in expression, pose, and gaze accuracy. Crucially, it scored the highest CSIM (Cosine Similarity of Identity), confirming its superior ability to preserve the source identity.

Qualitative Analysis

Qualitatively, MIMAFace excels at capturing nuanced facial details. It generates realistic skin wrinkles and light-shadow variations that provide a sense of depth and life. Unlike GAN-based methods that often warp at extreme angles, MIMAFace maintains a rational facial structure even in side profiles. The ICA module successfully removes the color shifts and flickering typically seen at clip boundaries in long-form generation.

Qualitative results demonstrating high-fidelity facial animation, nuanced expression details, and identity preservation.

Conclusion

MIMAFace represents a significant step forward in diffusion-based face animation. By bridging the gap between general texture and fine-grained identity through modulation, it provides a robust solution for high-fidelity talking head generation. While currently based on Stable Diffusion v1.5, future iterations utilizing more advanced base models like SDXL could further enhance image quality. As with all synthesis technology, we emphasize the importance of ethical use and the development of robust forgery detection methods.

Summary updated

Abstract

Current diffusion-based face animation methods generally adopt a ReferenceNet (a copy of U-Net) and a large amount of curated self-acquired data to learn appearance features, as robust appearance features are vital for ensuring temporal stability. However, when trained on public datasets, the results often exhibit a noticeable performance gap in image quality and temporal consistency. To address this issue, we meticulously examine the essential appearance features in the facial animation tasks, which include motion-agnostic (e.g., clothing, background) and motion-related (e.g., facial details) texture components, along with high-level discriminative identity features. Drawing from this analysis, we introduce a Motion-Identity Modulated Appearance Learning Module (MIA) that modulates CLIP features at both motion and identity levels. Additionally, to tackle the semantic/ color discontinuities between clips, we design an Inter-clip Affinity Learning Module (ICA) to model temporal relationships across clips. Our method achieves precise facial motion control (i.e., expressions and gaze), faithful identity preservation, and generates animation videos that maintain both intra/inter-clip temporal consistency. Moreover, it easily adapts to various modalities of driving sources. Extensive experiments demonstrate the superiority of our method.
BibTeX
@misc{han2024mimaface,
  author = {Yue Han and Junwei Zhu and Yuxiang Feng and Xiaozhong Ji and Keke He and Xiangtai Li and Zhucun Xue and Yong Liu},
  title = {{MIMAFace: Face Animation via Motion-Identity Modulated Appearance Feature Learning}},
  year = {2024},
  eprint = {2409.15179},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.15179},
}
Figure