Arc2Face: A Foundation Model of Human Faces

Foivos Paraperas Papantoniou, Alexandros Lattas, Stylianos Moschoglou, Jiankang Deng, Bernhard Kainz, Stefanos Zafeiriou

Introduction

Generating high-fidelity, identity-consistent human faces across diverse poses, expressions, and lighting conditions remains a significant challenge in computer vision. While modern text-to-image models like Stable Diffusion excel at general synthesis, they often suffer from "identity drift" when tasked with maintaining a specific person's features. This drift occurs because text prompts are inherently ambiguous for capturing fine-grained facial details, leading to a blend between the target identity and generic templates. Arc2Face addresses this by introducing an identity-conditioned foundation model that prioritizes facial embeddings over linguistic descriptions.

Methodology: The Identity-First Approach

Arc2Face shifts the generative paradigm by conditioning the diffusion process solely on identity embeddings. The framework utilizes ArcFace, a state-of-the-art face recognition model, to extract a compact 512-dimensional vector representing the unique "essence" of a face.

Overview of the Arc2Face pipeline, showing identity extraction and generation with optional spatial control via ControlNet.

To integrate these embeddings into a generative model, the researchers fine-tuned a CLIP text encoder to act as an adapter. By using a "pseudo-prompt" (e.g., "photo of a <id> person") where the <id> token is replaced by the ArcFace vector, the model leverages the pre-trained power of Stable Diffusion v1.5 while focusing exclusively on identity features. This design avoids the interference between language and identity that typically degrades performance in multi-modal models. Furthermore, Arc2Face can be combined with ControlNet to provide explicit spatial control over pose and expression using 3D face normals without compromising identity consistency.

Training at Scale

A core contribution of Arc2Face is the scale and quality of its training data. Recognizing that standard high-resolution datasets like FFHQ (70,000 images) lack the identity variety needed for a foundation model, the team utilized WebFace42M. They upsampled 21 million images to 512x512 resolution using GFPGAN face restoration. This massive dataset, encompassing 1 million unique identities, allows the model to learn a robust prior for the human facial manifold rather than simply memorizing specific training examples.

Experimental Results

Arc2Face demonstrates superior performance compared to existing state-of-the-art methods such as PhotoMaker and InstantID. Evaluation on benchmarks like Synth-500 and AgeDB highlights several key strengths:

  1. Identity Consistency: The model achieves the highest cosine similarity between input and generated faces, effectively eliminating identity drift.
  2. Diversity and Realism: With an LPIPS score of 0.506 (compared to InstantID's 0.386), Arc2Face generates significantly more diverse backgrounds, lighting, and hairstyles for the same subject. It also achieves a lower FID (5.673), indicating higher overall image realism.
  3. Synthetic Data Utility: Models trained solely on synthetic data generated by Arc2Face outperformed previous synthetic-only face recognition baselines, narrowing the gap with real-world data training.
Example of identity-consistent generation across various poses and expressions for a single subject.

Conclusion and Future Work

Arc2Face establishes a new baseline for identity-consistent face generation by treating identity as a primary condition rather than a secondary text attribute. While currently limited to single-subject generation and constrained by the discriminative power of the underlying ArcFace encoder, it provides a powerful foundation for tasks like 3D reconstruction and animation. The authors emphasize the importance of ethical responsibility in the deployment of such high-fidelity generators and support the development of robust deepfake detection tools.

Summary updated

BibTeX
@inproceedings{papantoniou2024arc,
  author = {Foivos Paraperas Papantoniou and Alexandros Lattas and Stylianos Moschoglou and Jiankang Deng and Bernhard Kainz and Stefanos Zafeiriou},
  title = {{Arc2Face: A Foundation Model of Human Faces}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72913-3\_14},
}
Figure