Abstract
Generating realistic talking head videos remains a challenge due to the difficulty of maintaining identity preservation and temporal consistency, especially when training on public datasets. We present MIMAFace, a novel diffusion-based framework that overcomes these hurdles. By modulating appearance features with motion and identity signals and introducing an inter-clip affinity module, MIMAFace generates high-fidelity animations with superior stability. Our approach effectively bridges the gap between low-level generative features and high-level identity markers, outperforming current state-of-the-art methods in motion accuracy and identity preservation.
Introduction
Recent advancements in diffusion models have pushed the boundaries of face animation. Most existing methods utilize a "ReferenceNet" (a duplicated U-Net) to maintain the appearance of a source image. However, these models often struggle with two primary issues: the Data Gap and Temporal Discontinuity.
When trained on public datasets like VoxCeleb, current models frequently produce stiff expressions or lose the subject's identity. Furthermore, generating long-form video by stitching short clips often results in jarring "jumps" in lighting or semantics. MIMAFace addresses these by arguing that models must capture motion-related textures (e.g., facial wrinkles) and high-level identity markers more effectively than standard CLIP-based appearance encoders allow.
Methodology
MIMAFace introduces two core innovations to the standard diffusion pipeline: Motion-Identity Modulated Appearance Learning (MIA) and Inter-clip Affinity Learning (ICA).
Motion-Identity Modulated Appearance Learning (MIA)
Rather than relying solely on raw CLIP features, the MIA module modulates them using dynamic signals:
- Motion Modulation: By incorporating 3D Morphable Model (3DMM) coefficients and gaze embeddings, the model learns how textures evolve with movement, such as skin folding during a smile.
- Identity Modulation: To ensure the subject remains recognizable throughout the denoising process, we introduce an Identity Contrastive Loss ($L_{id}$). This forces the encoder to prioritize discriminative features that are invariant to lighting or pose changes.
Inter-clip Affinity Learning (ICA)
To eliminate flickering between video segments, the ICA module models the relationship between the current clip and the preceding one. By conditioning the model on the latent representations of previous frames during training, we ensure that new clips begin as a seamless continuation of the last, maintaining temporal flow across long sequences.
Experiments and Datasets
The model was trained on VoxCeleb2 and evaluated on VoxCeleb1. To test generalization and robustness, we utilized FFHQ for unseen identities, MPIE for extreme head poses, and VFHQ for high-definition (768x768) synthesis.
Key Results
Quantitative Performance
MIMAFace demonstrates state-of-the-art performance on the VoxCeleb1 benchmark, outperforming popular methods like AniPortrait and FADM. Specifically, it achieved the lowest error rates in expression, pose, and gaze accuracy. Crucially, it scored the highest CSIM (Cosine Similarity of Identity), confirming its superior ability to preserve the source identity.
Qualitative Analysis
Qualitatively, MIMAFace excels at capturing nuanced facial details. It generates realistic skin wrinkles and light-shadow variations that provide a sense of depth and life. Unlike GAN-based methods that often warp at extreme angles, MIMAFace maintains a rational facial structure even in side profiles. The ICA module successfully removes the color shifts and flickering typically seen at clip boundaries in long-form generation.
Conclusion
MIMAFace represents a significant step forward in diffusion-based face animation. By bridging the gap between general texture and fine-grained identity through modulation, it provides a robust solution for high-fidelity talking head generation. While currently based on Stable Diffusion v1.5, future iterations utilizing more advanced base models like SDXL could further enhance image quality. As with all synthesis technology, we emphasize the importance of ethical use and the development of robust forgery detection methods.
Summary updated