Introduction
Generating high-fidelity, identity-consistent human faces across diverse poses, expressions, and lighting conditions remains a significant challenge in computer vision. While modern text-to-image models like Stable Diffusion excel at general synthesis, they often suffer from "identity drift" when tasked with maintaining a specific person's features. This drift occurs because text prompts are inherently ambiguous for capturing fine-grained facial details, leading to a blend between the target identity and generic templates. Arc2Face addresses this by introducing an identity-conditioned foundation model that prioritizes facial embeddings over linguistic descriptions.
Methodology: The Identity-First Approach
Arc2Face shifts the generative paradigm by conditioning the diffusion process solely on identity embeddings. The framework utilizes ArcFace, a state-of-the-art face recognition model, to extract a compact 512-dimensional vector representing the unique "essence" of a face.
To integrate these embeddings into a generative model, the researchers fine-tuned a CLIP text encoder to act as an adapter. By using a "pseudo-prompt" (e.g., "photo of a <id> person") where the <id> token is replaced by the ArcFace vector, the model leverages the pre-trained power of Stable Diffusion v1.5 while focusing exclusively on identity features. This design avoids the interference between language and identity that typically degrades performance in multi-modal models. Furthermore, Arc2Face can be combined with ControlNet to provide explicit spatial control over pose and expression using 3D face normals without compromising identity consistency.
Training at Scale
A core contribution of Arc2Face is the scale and quality of its training data. Recognizing that standard high-resolution datasets like FFHQ (70,000 images) lack the identity variety needed for a foundation model, the team utilized WebFace42M. They upsampled 21 million images to 512x512 resolution using GFPGAN face restoration. This massive dataset, encompassing 1 million unique identities, allows the model to learn a robust prior for the human facial manifold rather than simply memorizing specific training examples.
Experimental Results
Arc2Face demonstrates superior performance compared to existing state-of-the-art methods such as PhotoMaker and InstantID. Evaluation on benchmarks like Synth-500 and AgeDB highlights several key strengths:
- Identity Consistency: The model achieves the highest cosine similarity between input and generated faces, effectively eliminating identity drift.
- Diversity and Realism: With an LPIPS score of 0.506 (compared to InstantID's 0.386), Arc2Face generates significantly more diverse backgrounds, lighting, and hairstyles for the same subject. It also achieves a lower FID (5.673), indicating higher overall image realism.
- Synthetic Data Utility: Models trained solely on synthetic data generated by Arc2Face outperformed previous synthetic-only face recognition baselines, narrowing the gap with real-world data training.
Conclusion and Future Work
Arc2Face establishes a new baseline for identity-consistent face generation by treating identity as a primary condition rather than a secondary text attribute. While currently limited to single-subject generation and constrained by the discriminative power of the underlying ArcFace encoder, it provides a powerful foundation for tasks like 3D reconstruction and animation. The authors emphasize the importance of ethical responsibility in the deployment of such high-fidelity generators and support the development of robust deepfake detection tools.
Summary updated