IDiff-Face: Synthetic-based Face Recognition through Fizzy Identity-Conditioned Diffusion Model

Fadi Boutros, Jonas Henry Grebe, Arjan Kuijper, Naser DamerICCV 2023

TL;DR

This paper introduces IDiff-Face, a novel approach for generating synthetic face images using identity-conditioned diffusion models. By addressing the limitations of existing synthetic datasets in terms of identity discrimination and intra-class diversity, IDiff-Face achieves state-of-the-art performance in synthetic-based face recognition, significantly closing the gap with models trained on authentic data.

What Problem Does This Paper Address?

The field of face recognition (FR) has seen remarkable progress, largely fueled by the availability of massive, authentic face datasets. However, these datasets raise significant privacy concerns, leading to their retraction and posing a challenge to the continuity of FR research. While synthetic data has emerged as a promising alternative, current synthetic datasets often suffer from a trade-off: either they have good identity discrimination but lack sufficient intra-class variation, or vice versa. This imbalance leads to suboptimal performance for FR models trained on them, falling short of accuracies achieved with authentic data. This paper aims to bridge this gap by generating synthetic face images that possess both strong identity separability and desirable intra-class diversity, making them suitable for training high-performance FR models.

Methods

The core of IDiff-Face is an identity-conditioned latent diffusion model (DM). The process involves two main stages:

  1. Training: The diffusion model is trained in the latent space of a pre-trained autoencoder (AE). Crucially, the diffusion model is conditioned on "identity contexts," which are feature representations extracted from authentic training images using a pre-trained face recognition model. This conditioning is achieved through a Cross-Attention (CA) mechanism.
  2. Sampling: To generate synthetic images, the trained DM can utilize different types of identity contexts: "uniform" (sampling identity representations from a distribution) or "two-stage" (generating an initial synthetic image with an unconditional DM and then using its features as context).

A key innovation is the Contextual Partial Dropout (CPD) technique. This method introduces a "fizzy" aspect by partially dropping components of the identity context during training. This prevents the model from overfitting to the identity context, thereby encouraging larger intra-class variations and creating a more realistic trade-off between identity discrimination and diversity.

Datasets

  • Training: The FFHQ dataset (70,000 high-quality face images) was used to train the diffusion model.
  • Evaluation: The paper evaluates its approach on several standard face recognition benchmarks, including:
  • Labeled Faces in the Wild (LFW)
    • AgeDB-30
    • CA-LFW
    • CFP-FP
    • CP-LFW

The paper also compares its generated synthetic datasets against existing SOTA synthetic datasets like DiscoFaceGAN, SFace, and DigiFace-1M.

Key Insights and Ideas

  • The "Fizzy" Trade-off: The core insight is that achieving high FR accuracy requires a delicate balance between identity discrimination and intra-class variation. Existing synthetic datasets struggle with this balance.
  • CPD for Controlled Diversity: Contextual Partial Dropout (CPD) is a simple yet effective mechanism to control this trade-off, allowing for more realistic variations in generated images without sacrificing too much identity separability.
  • Diffusion Models for Synthetic Identity Generation: Diffusion models, known for their stability and high-quality image generation, are well-suited for creating diverse synthetic identities when conditioned appropriately.
  • Bridging the Gap: The research demonstrates that high-quality synthetic data can indeed rival authentic data for FR training, significantly reducing reliance on privacy-sensitive datasets.

Key Results

  • LFW Performance: IDiff-Face achieved 98.00% accuracy on LFW, surpassing previous synthetic-based FR solutions (e.g., 95.40%) and approaching authentic-based FR (99.82%). This also exceeds human-level performance on LFW (97.5%).
  • SOTA Synthetic-Based FR: When trained on 500K images from 10K identities, IDiff-Face achieved an average accuracy of 88.20%, significantly outperforming the best previous SOTA synthetic-based FR (83.45%).
  • Dataset Size Impact: Increasing dataset width (number of identities) was found to be more beneficial than increasing dataset depth (images per identity) for improving FR accuracy.
  • CPD Effectiveness: Experiments with different CPD probabilities (0%, 25%, 50%) showed that higher CPD levels increased intra-class variation, leading to improved FR accuracies, albeit with a slight decrease in identity discrimination.

Limitations and Future Work

  • Computational Cost: While diffusion models are generally easier to train than GANs, generating large-scale synthetic datasets can still be computationally intensive.
  • Further Optimization: The paper suggests that further improvements in verification accuracy could be achieved by exploring larger synthetic training datasets and more advanced data augmentation techniques.
  • Generalizability: While the focus is on FR, the principles of identity-conditioned diffusion models with controlled diversity could potentially be applied to other generative tasks requiring identity preservation.

Summary updated

BibTeX
@inproceedings{boutros2023idiff,
  author = {Fadi Boutros and Jonas Henry Grebe and Arjan Kuijper and Naser Damer},
  title = {{IDiff-Face: Synthetic-based Face Recognition through Fizzy Identity-Conditioned Diffusion Model}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure