CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition

Zhonglin Sun, Siyang Song, Ioannis Patras, Georgios Tzimiropoulos

Training state-of-the-art face recognition (FR) models typically requires millions of real-world images. However, with the rise of strict privacy regulations like GDPR and ethical concerns over non-consensual data scraping, the "data hunger" of deep learning is hitting a legal and ethical wall. Synthetic data offers a potential solution, but until now, FR models trained on synthetic faces have consistently underperformed compared to those trained on real data.

In their NeurIPS 2024 paper, researchers from Queen Mary University of London and the University of Exeter propose CemiFace, a diffusion-based generative model that bridges this performance gap by rethinking what makes a synthetic image effective for training.

The "Gap-to-Real" Challenge

The core challenge in Synthetic Face Recognition (SFR) is the "Gap-to-Real" error. Even when generating high-quality images using tools like StyleGAN, the resulting FR models are often significantly less accurate than those trained on real datasets like CASIA-WebFace. The authors argue that existing generative methods focus too much on visual quality and not enough on discriminative quality.

In FR training, "easy" samples (clear, frontal shots) don't challenge the model enough, while "extreme" samples (heavy occlusion or noise) can prevent convergence. The ideal training data consists of semi-hard samples: images that belong to an identity but are different enough from the "average" face of that person to force the model to learn robust features.

Methodology: Targeting the Semi-Hard Zone

CemiFace is a conditional diffusion model designed to generate images at specific "radii" from an identity center in the latent feature space. The authors' preliminary experiments on real data suggested that samples with mid-level similarity (~0.7 cosine similarity) produced the best-performing models.

Conceptual overview of CemiFace: generating samples at varying similarity levels (radii) from an identity center.

Key Technical Innovations

  1. Similarity Controlling Factor ($m$): CemiFace introduces a scalar $m$ (ranging from -1 to 1) that explicitly regulates how similar the output should be to an "inquiry image" (the identity anchor).
  2. Similarity Matching Loss ($L_{SimMat}$): During training, the model uses a pre-trained FR model to measure the similarity between the generated image and the inquiry image. A loss function then forces the generator to respect the requested similarity $m$.
  3. Cross-Attention Conditioning: The model injects both the identity embedding and the similarity factor $m$ into the Diffusion UNet using cross-attention layers, allowing for fine-grained control over the generation process.

Key Insights and Results

The most effective training happens when $m \approx 0$. At this level, the images look like the same person but exhibit significant variations in pose, age, and lighting—the exact "semi-hard" challenges a model needs to become robust. Unlike previous methods that focused on transferring "styles" (hair, lighting) from a style bank, CemiFace shows that explicitly controlling similarity is a more direct and effective way to create discriminative training samples.

Examples of generated semi-hard samples with varying similarity scores, demonstrating the diversity in pose and expression while maintaining identity.

CemiFace sets a new state-of-the-art for synthetic face recognition:

  • Accuracy Boost: On a 0.5M image training protocol, CemiFace achieved an average accuracy of 92.30% across benchmarks (LFW, CFP-FP, AgeDB-30, CPLFW, and CALFW), significantly outperforming previous leaders like DCFace (89.56%).
  • Halving the Gap: It reduced the "Gap-to-Real" error by 51.96% relative to previous methods.
  • Pose Robustness: The model showed particularly strong gains on pose-sensitive datasets like CFP-FP, improving accuracy by over 5%.

Conclusion

CemiFace demonstrates that the path to better synthetic data isn't just about making images look more "real"—it's about making them "hard" in exactly the right way. Because CemiFace anchors its generation on an inquiry image rather than a fixed set of class labels, it can theoretically generate an infinite number of identities from any collection of "in-the-wild" face images, providing a scalable and privacy-conscious path forward for face recognition research.

Summary updated

BibTeX
@inproceedings{sun2024cemiface,
  author = {Zhonglin Sun and Siyang Song and Ioannis Patras and Georgios Tzimiropoulos},
  title = {{CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=ykQnxko1cJ},
}
Figure