Recoverable Facial Identity Protection via Adaptive Makeup Transfer Adversarial Attacks

Xiyao Liu, Junxing Ma, Xinda Wang, Qianyu Lin, Jian Zhang, Gerald Schaefer, Cagatay Turkay, Hui FangAAAI 2025 · DOI

Abstract

Unauthorized face recognition (FR) systems pose a significant threat to personal privacy through the automated scraping of social media imagery. While adversarial attacks can mitigate this risk, existing methods often suffer from extreme visual distortion or irreversibility, rendering protected images useless for authorized applications. We present RMT-GAN (Recoverable Makeup Transferred Generative Adversarial Network), a framework that applies natural-looking adversarial makeup to protect facial identity while maintaining a "backdoor" for authorized recovery. By adaptively selecting target identities and employing a cycle-consistent recovery module, RMT-GAN achieves high attack success rates and superior visual quality compared to state-of-the-art methods.

Introduction

As face recognition technology becomes ubiquitous, the need for effective privacy-preserving tools has grown. Adversarial attacks—subtle pixel perturbations or digital "makeup"—can prevent unauthorized systems from identifying users. However, current makeup-based attacks face two primary hurdles: visual distortion and irreversibility. Many methods force a user's face to match a fixed, unrelated target identity, leading to unnatural artifacts and "domain conflict." Furthermore, once an adversarial photo is uploaded, it remains "broken" for everyone, including authorized services like secure payment check-ins.

Methodology

RMT-GAN addresses these issues through a three-component architecture: a Generator ($G$), a Discriminator ($D$), and a dedicated Recovery module ($R$).

Adaptive Target Selection

To minimize visual distortion, RMT-GAN avoids forcing a user's face toward a random target. Instead, the system identifies the top 20% most geometrically similar faces in a reference dataset. By selecting a "neighbor" as the target identity, the generator requires less warping to achieve misidentification, resulting in significantly higher image quality and naturalness.

Adversarial makeup generated by RMT-GAN to protect identity.

The Recovery Module

The core innovation of RMT-GAN is its reversibility. During training, the model utilizes a cycle consistency loss specifically designed for recovery. While the generator learns to embed the adversarial signal through makeup transfer, the recovery module learns to remove it. This ensures that the transformation is mathematically reversible, allowing authorized parties who possess the recovery module to "strip" the makeup and restore the original identity.

Experimental Results

The researchers evaluated RMT-GAN using the Makeup Transfer (MT), CelebA-HQ, and LADN datasets, alongside commercial black-box APIs like Face++ and Aliyun.

Attack Performance and Visual Quality

RMT-GAN significantly outperformed state-of-the-art (SoTA) methods. On the CelebA-HQ dataset, it achieved an average Attack Success Rate (ASR) of 66.9%, compared to 44.9% for AMT-GAN and 12.5% for Adv-Makeup. In a user study with 25 participants, RMT-GAN's images were rated as superior in naturalness and quality 88% of the time.

Successful Identity Recovery

The recovery module proved highly effective in restoring utility for authorized systems. For instance, while the adversarial attack reduced recognition rates on the MobileFace model to nearly zero, the recovery module restored it to 17.2%, very close to the original source's 19.4% baseline.

The original source image, which can be restored via the recovery module.

Conclusion

RMT-GAN transforms adversarial attacks from destructive processes into functional privacy tools. By prioritizing visual naturalness and recoverability, it provides a robust defense against unauthorized face recognition without sacrificing the utility of photos for legitimate, authorized applications. Future work will focus on more localized makeup strategies and bridging the gap between digital transfer and physical-world application.

Summary updated

BibTeX
@inproceedings{liu2025recoverable,
  author = {Xiyao Liu and Junxing Ma and Xinda Wang and Qianyu Lin and Jian Zhang and Gerald Schaefer and Cagatay Turkay and Hui Fang},
  title = {{Recoverable Facial Identity Protection via Adaptive Makeup Transfer Adversarial Attacks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i1.32031},
}
Figure