Mitigating the Impact of Attribute Editing on Face Recognition

Sudipta Banerjee, Sai Pranaswi Mullangi, Shruti Wagle, Chinmay Hegde, Nasir D. Memon

Abstract

Through a large-scale study over diverse face images, we show that facial attribute editing using modern generative AI models can severely degrade automated face recognition systems. This degradation persists even with identity-preserving generative models. To mitigate this issue, we propose two novel techniques for local and global attribute editing. We empirically ablate twenty-six facial semantic, demographic and expression-based attributes that have been edited using state-of-the-art generative models, and evaluate them using ArcFace and AdaFace matchers on CelebA, CelebAMaskHQ and LFW datasets. Finally, we use LLaVA, an emerging visual question-answering framework for attribute prediction to validate our editing techniques. Our methods outperform the current state-of-the-art at facial editing (BLIP, InstantID) while improving identity retention by a significant extent.

Introduction

Facial attribute editing has transitioned from professional retouching to consumer-grade applications powered by sophisticated diffusion models. While these tools produce visually realistic results, they often fail to maintain "biometric fidelity"—the specific features required for automated face recognition (FR) systems to identify an individual. This creates a security gap where simple digital edits can inadvertently act as "evasion attacks," undermining the reliability of identity verification systems. This research investigates the extent of this biometric degradation and proposes two identity-preserving editing techniques.

Methodology

The researchers categorize facial editing into two levels, proposing distinct mitigation strategies for each.

Global Attribute Editing (DB-prop)

For coarse changes affecting the entire face (e.g., age, gender), the authors introduce DB-prop, a modified version of DreamBooth. The core innovation is a contrastive loss function that anchors the model to the subject's unique identity. By comparing the target individual against a regularization set of diverse faces, the model learns to modify specific attributes while preserving the underlying biometric "essence."

Overview of the global attribute editing framework (DB-prop) using contrastive loss and regularization sets to preserve identity.

Local Attribute Editing (CN-IP)

For fine-grained changes (e.g., adding glasses or changing hair color), the authors propose CN-IP, which utilizes ControlNet with Inpainting. This method employs semantic segmentation masks to define the edit region and depth maps to preserve the face's 3D geometry. This structural blueprint ensures that identity-critical features remain unchanged even within the edited area.

Local attribute editing using masks to constrain modifications to specific facial regions.

To automate evaluation, the study pioneered the use of LLaVA (Large Language-and-Vision Assistant) to verify attribute changes, providing a scalable and objective alternative to traditional human user studies.

Datasets

The study utilized several benchmark datasets to ensure robust evaluation:

  • CelebA & CelebAMaskHQ: Used for training and local editing evaluation.
  • LFW (Labeled Faces in the Wild): Used for testing face recognition in unconstrained environments.
  • AffectNet: Sourced for diverse facial expressions in regularization sets.

Key Results and Insights

The proposed methods demonstrated significant improvements in identity retention compared to baseline generative models:

  • Superior Biometric Preservation: For global editing on the LFW dataset, DB-prop outperformed the standard DreamBooth baseline by 73% (ArcFace) and 17% (AdaFace).
  • Local Precision: The CN-IP method surpassed BLIP-Diffusion by 74% in identity retention while successfully executing local edits.
  • Vulnerability Analysis: The study found that certain edits, such as changing hair color to black or making a subject bald, are particularly destructive to recognition systems, causing False Non-Match Rates (FNMR) to skyrocket in baseline models.
  • Structure vs. Style: Preserving 3D structure via depth maps proved more effective for identity retention than using 2D edges (Canny maps).

Conclusion and Future Work

While the proposed methods significantly mitigate biometric degradation, challenges remain. Editing complex facial expressions (e.g., "angry") can still lead to "hallucinations" that compromise identity. Additionally, the computational cost of fine-tuning for global editing (10-15 minutes per subject) currently limits real-time application. Future research will focus on "tuning-free" methods to achieve similar biometric stability with greater efficiency.

Summary updated

Abstract

Through a large-scale study over diverse face images, we show that facial attribute editing using modern generative AI models can severely degrade automated face recognition systems. This degradation persists even with identity-preserving generative models. To mitigate this issue, we propose two novel techniques for local and global attribute editing. We empirically ablate twenty-six facial semantic, demographic and expression-based attributes that have been edited using state-of-the-art generative models, and evaluate them using ArcFace and AdaFace matchers on CelebA, CelebAMaskHQ and LFW datasets. Finally, we use LLaVA, an emerging visual question-answering framework for attribute prediction to validate our editing techniques. Our methods outperform the current state-of-the-art at facial editing (BLIP, InstantID) while improving identity retention by a significant extent.
BibTeX
@inproceedings{banerjee2024mitigating,
  author = {Sudipta Banerjee and Sai Pranaswi Mullangi and Shruti Wagle and Chinmay Hegde and Nasir D. Memon},
  title = {{Mitigating the Impact of Attribute Editing on Face Recognition}},
  booktitle = {IJCB},
  year = {2024},
  doi = {10.1109/ijcb62174.2024.10744430},
}
Figure