Identity-Preserving Aging of Face Images via Latent Diffusion Models

Sudipta Banerjee, Govind Mittal, Ameya Joshi, Chinmay Hegde, Nasir D. Memon

TL;DR

Researchers from NYU have developed a method to synthetically age and de-age faces using Latent Diffusion Models (LDMs) while strictly preserving the individual's unique biometric identity. By fine-tuning models like Stable Diffusion with a novel combination of biometric and contrastive losses, they achieved a ~44% reduction in False Non-Match Rates compared to previous state-of-the-art GAN-based methods.

What Problem Does This Paper Address?

Automated face recognition (AFR) systems are notoriously sensitive to facial aging. As we age, our features change in complex ways influenced by genetics, lifestyle, and environment, often causing AFR systems to fail when comparing a current photo to an older enrollment image.

The biggest hurdle in solving this is the lack of "longitudinal data"—high-quality datasets showing the same individuals over many decades. While Generative Adversarial Networks (GANs) have been used to "simulate" aging, they often struggle with a fundamental trade-off: they either change the age but lose the person's identity, or they keep the identity but fail to produce a realistic aging effect. Furthermore, GANs typically require massive amounts of data to train effectively.

Methods

The authors build upon DreamBooth, a technique for fine-tuning text-to-image diffusion models to recognize a specific subject using only a few images (~20). Their approach, however, adds two critical components to ensure the person still looks like themselves after "aging":

  1. Identity-Preserving Losses: They integrated a biometric loss (using a pre-trained VGGFace extractor) and a contrastive loss (based on the SimCLR framework). These losses penalize the model if the generated "old" version of a person deviates too much from the biometric "fingerprint" of the original images.
  2. Age-Aware Regularization: To teach the model what "aging" actually looks like without it forgetting the specific person, they used a small auxiliary "regularization set" of about 600 image-caption pairs. These pairs categorize faces into six groups: child, teenager, youngadults, middleaged, elderly, and old.
  3. Text-Guided Inference: Once trained, the model is controllable via simple text prompts. By using a unique identifier (a "rare token" like sks) associated with the subject, a user can prompt: "photo of a sks person as elderly" to generate the aged version.

Datasets

The study utilized several key benchmarks:

  • CelebA & AgeDB: Used as the primary "training sets" for the target individuals (100 subjects from each).
  • CelebA-Dialog: Used to create the Regularization Set, providing the fine-grained age distributions needed to teach the model the concept of age progression.
  • CACD (Cross-Age Celebrity Dataset): Used for comparative evaluation against existing baselines.

Key Insights and Ideas

  • Few-Shot Power: Unlike GANs that need thousands of images, this diffusion-based approach succeeds with as few as 20 images of a target individual.
  • Controllability via Language: Because the model is built on a text-to-image backbone, aging is "promptable." You don't need complex attribute vectors; you just tell the model the target age group in plain English.
  • Synthetic Data as a Teacher: A fascinating finding was that face recognition classifiers actually performed better after being fine-tuned on these synthetically aged images. This suggests that the model generates variations that are realistic enough to help "harden" AFR systems against real-world aging.
  • Rare Tokens Matter: The use of specific rare tokens (like wzx or sks) is crucial. These tokens act as a "hook" in the model's latent space, allowing it to bind a specific identity to the general concept of a "person."

Key Results

  • Biometric Fidelity: The method significantly outperformed baselines like IPCGAN and AttGAN. On the CelebA dataset, it achieved a 44% average reduction in the False Non-Match Rate (FNMR) compared to Talk-to-Edit.
  • User Study: In a study with 26 volunteers, participants were able to correctly identify the original subject in the generated images with high accuracy, and they correctly categorized the intended age groups (e.g., 99.6% accuracy for the 'child' group).
  • Visual Realism: Qualitatively, the images show much higher resolution and more natural textural changes (wrinkles, skin tone shifts) than previous GAN-based approaches, which often produced "blurry" or "ghostly" artifacts.

Limitations and Future Work

Despite the success, the authors noted a few "failure cases":

  • Extreme De-aging: The model sometimes struggles to generate "child" versions of individuals if the training set consists entirely of elderly people or people with heavy facial hair.
  • Demographic Biases: There are slight variations in performance across different sexes and ethnicities, likely inherited from the underlying pre-trained Stable Diffusion model. For instance, the model occasionally adds makeup when aging women compared to men.
  • Future Directions: The team aims to explore zero-shot age editing (aging a person without needing any fine-tuning images at all) and using "composable" diffusion models for even finer control over specific facial features during the aging process.

Summary updated

Abstract

The performance of automated face recognition systems is inevitably impacted by the facial aging process. However, high quality datasets of individuals collected over several years are typically small in scale. In this work, we propose, train, and validate the use of latent text-to-image diffusion models for synthetically aging and de-aging face images. Our models succeed with few-shot training, and have the added benefit of being controllable via intuitive textual prompting. We observe high degrees of visual realism in the generated images while maintaining biometric fidelity measured by commonly used metrics. We evaluate our method on two benchmark datasets (CelebA and AgeDB) and observe significant reduction (~ 44%) in the False Non-Match Rate compared to existing state-of the-art baselines.
BibTeX
@inproceedings{banerjee2023identity,
  author = {Sudipta Banerjee and Govind Mittal and Ameya Joshi and Chinmay Hegde and Nasir D. Memon},
  title = {{Identity-Preserving Aging of Face Images via Latent Diffusion Models}},
  booktitle = {IJCB},
  year = {2023},
  url = {https://openreview.net/forum?id=wEXKmj7pfH},
}

Loading related content...

Figure