Abstract
Through a large-scale study over diverse face images, we show that facial attribute editing using modern generative AI models can severely degrade automated face recognition systems. This degradation persists even with identity-preserving generative models. To mitigate this issue, we propose two novel techniques for local and global attribute editing. We empirically ablate twenty-six facial semantic, demographic and expression-based attributes that have been edited using state-of-the-art generative models, and evaluate them using ArcFace and AdaFace matchers on CelebA, CelebAMaskHQ and LFW datasets. Finally, we use LLaVA, an emerging visual question-answering framework for attribute prediction to validate our editing techniques. Our methods outperform the current state-of-the-art at facial editing (BLIP, InstantID) while improving identity retention by a significant extent.
Introduction
Facial attribute editing has transitioned from professional retouching to consumer-grade applications powered by sophisticated diffusion models. While these tools produce visually realistic results, they often fail to maintain "biometric fidelity"—the specific features required for automated face recognition (FR) systems to identify an individual. This creates a security gap where simple digital edits can inadvertently act as "evasion attacks," undermining the reliability of identity verification systems. This research investigates the extent of this biometric degradation and proposes two identity-preserving editing techniques.
Methodology
The researchers categorize facial editing into two levels, proposing distinct mitigation strategies for each.
Global Attribute Editing (DB-prop)
For coarse changes affecting the entire face (e.g., age, gender), the authors introduce DB-prop, a modified version of DreamBooth. The core innovation is a contrastive loss function that anchors the model to the subject's unique identity. By comparing the target individual against a regularization set of diverse faces, the model learns to modify specific attributes while preserving the underlying biometric "essence."
Local Attribute Editing (CN-IP)
For fine-grained changes (e.g., adding glasses or changing hair color), the authors propose CN-IP, which utilizes ControlNet with Inpainting. This method employs semantic segmentation masks to define the edit region and depth maps to preserve the face's 3D geometry. This structural blueprint ensures that identity-critical features remain unchanged even within the edited area.
To automate evaluation, the study pioneered the use of LLaVA (Large Language-and-Vision Assistant) to verify attribute changes, providing a scalable and objective alternative to traditional human user studies.
Datasets
The study utilized several benchmark datasets to ensure robust evaluation:
- CelebA & CelebAMaskHQ: Used for training and local editing evaluation.
- LFW (Labeled Faces in the Wild): Used for testing face recognition in unconstrained environments.
- AffectNet: Sourced for diverse facial expressions in regularization sets.
Key Results and Insights
The proposed methods demonstrated significant improvements in identity retention compared to baseline generative models:
- Superior Biometric Preservation: For global editing on the LFW dataset, DB-prop outperformed the standard DreamBooth baseline by 73% (ArcFace) and 17% (AdaFace).
- Local Precision: The CN-IP method surpassed BLIP-Diffusion by 74% in identity retention while successfully executing local edits.
- Vulnerability Analysis: The study found that certain edits, such as changing hair color to black or making a subject bald, are particularly destructive to recognition systems, causing False Non-Match Rates (FNMR) to skyrocket in baseline models.
- Structure vs. Style: Preserving 3D structure via depth maps proved more effective for identity retention than using 2D edges (Canny maps).
Conclusion and Future Work
While the proposed methods significantly mitigate biometric degradation, challenges remain. Editing complex facial expressions (e.g., "angry") can still lead to "hallucinations" that compromise identity. Additionally, the computational cost of fine-tuning for global editing (10-15 minutes per subject) currently limits real-time application. Future research will focus on "tuning-free" methods to achieve similar biometric stability with greater efficiency.
Summary updated