TL;DR
Researchers from NYU have developed a method to synthetically age and de-age faces using Latent Diffusion Models (LDMs) while strictly preserving the individual's unique biometric identity. By fine-tuning models like Stable Diffusion with a novel combination of biometric and contrastive losses, they achieved a ~44% reduction in False Non-Match Rates compared to previous state-of-the-art GAN-based methods.
What Problem Does This Paper Address?
Automated face recognition (AFR) systems are notoriously sensitive to facial aging. As we age, our features change in complex ways influenced by genetics, lifestyle, and environment, often causing AFR systems to fail when comparing a current photo to an older enrollment image.
The biggest hurdle in solving this is the lack of "longitudinal data"—high-quality datasets showing the same individuals over many decades. While Generative Adversarial Networks (GANs) have been used to "simulate" aging, they often struggle with a fundamental trade-off: they either change the age but lose the person's identity, or they keep the identity but fail to produce a realistic aging effect. Furthermore, GANs typically require massive amounts of data to train effectively.
Methods
The authors build upon DreamBooth, a technique for fine-tuning text-to-image diffusion models to recognize a specific subject using only a few images (~20). Their approach, however, adds two critical components to ensure the person still looks like themselves after "aging":
- Identity-Preserving Losses: They integrated a biometric loss (using a pre-trained VGGFace extractor) and a contrastive loss (based on the SimCLR framework). These losses penalize the model if the generated "old" version of a person deviates too much from the biometric "fingerprint" of the original images.
- Age-Aware Regularization: To teach the model what "aging" actually looks like without it forgetting the specific person, they used a small auxiliary "regularization set" of about 600 image-caption pairs. These pairs categorize faces into six groups: child, teenager, youngadults, middleaged, elderly, and old.
- Text-Guided Inference: Once trained, the model is controllable via simple text prompts. By using a unique identifier (a "rare token" like
sks) associated with the subject, a user can prompt: "photo of a sks person as elderly" to generate the aged version.
Datasets
The study utilized several key benchmarks:
- CelebA & AgeDB: Used as the primary "training sets" for the target individuals (100 subjects from each).
- CelebA-Dialog: Used to create the Regularization Set, providing the fine-grained age distributions needed to teach the model the concept of age progression.
- CACD (Cross-Age Celebrity Dataset): Used for comparative evaluation against existing baselines.
Key Insights and Ideas
- Few-Shot Power: Unlike GANs that need thousands of images, this diffusion-based approach succeeds with as few as 20 images of a target individual.
- Controllability via Language: Because the model is built on a text-to-image backbone, aging is "promptable." You don't need complex attribute vectors; you just tell the model the target age group in plain English.
- Synthetic Data as a Teacher: A fascinating finding was that face recognition classifiers actually performed better after being fine-tuned on these synthetically aged images. This suggests that the model generates variations that are realistic enough to help "harden" AFR systems against real-world aging.
- Rare Tokens Matter: The use of specific rare tokens (like
wzxorsks) is crucial. These tokens act as a "hook" in the model's latent space, allowing it to bind a specific identity to the general concept of a "person."
Key Results
- Biometric Fidelity: The method significantly outperformed baselines like IPCGAN and AttGAN. On the CelebA dataset, it achieved a 44% average reduction in the False Non-Match Rate (FNMR) compared to Talk-to-Edit.
- User Study: In a study with 26 volunteers, participants were able to correctly identify the original subject in the generated images with high accuracy, and they correctly categorized the intended age groups (e.g., 99.6% accuracy for the 'child' group).
- Visual Realism: Qualitatively, the images show much higher resolution and more natural textural changes (wrinkles, skin tone shifts) than previous GAN-based approaches, which often produced "blurry" or "ghostly" artifacts.
Limitations and Future Work
Despite the success, the authors noted a few "failure cases":
- Extreme De-aging: The model sometimes struggles to generate "child" versions of individuals if the training set consists entirely of elderly people or people with heavy facial hair.
- Demographic Biases: There are slight variations in performance across different sexes and ethnicities, likely inherited from the underlying pre-trained Stable Diffusion model. For instance, the model occasionally adds makeup when aging women compared to men.
- Future Directions: The team aims to explore zero-shot age editing (aging a person without needing any fine-tuning images at all) and using "composable" diffusion models for even finer control over specific facial features during the aging process.
Summary updated