Abstract
Unauthorized face recognition (FR) systems pose a significant threat to personal privacy through the automated scraping of social media imagery. While adversarial attacks can mitigate this risk, existing methods often suffer from extreme visual distortion or irreversibility, rendering protected images useless for authorized applications. We present RMT-GAN (Recoverable Makeup Transferred Generative Adversarial Network), a framework that applies natural-looking adversarial makeup to protect facial identity while maintaining a "backdoor" for authorized recovery. By adaptively selecting target identities and employing a cycle-consistent recovery module, RMT-GAN achieves high attack success rates and superior visual quality compared to state-of-the-art methods.
Introduction
As face recognition technology becomes ubiquitous, the need for effective privacy-preserving tools has grown. Adversarial attacks—subtle pixel perturbations or digital "makeup"—can prevent unauthorized systems from identifying users. However, current makeup-based attacks face two primary hurdles: visual distortion and irreversibility. Many methods force a user's face to match a fixed, unrelated target identity, leading to unnatural artifacts and "domain conflict." Furthermore, once an adversarial photo is uploaded, it remains "broken" for everyone, including authorized services like secure payment check-ins.
Methodology
RMT-GAN addresses these issues through a three-component architecture: a Generator ($G$), a Discriminator ($D$), and a dedicated Recovery module ($R$).
Adaptive Target Selection
To minimize visual distortion, RMT-GAN avoids forcing a user's face toward a random target. Instead, the system identifies the top 20% most geometrically similar faces in a reference dataset. By selecting a "neighbor" as the target identity, the generator requires less warping to achieve misidentification, resulting in significantly higher image quality and naturalness.
The Recovery Module
The core innovation of RMT-GAN is its reversibility. During training, the model utilizes a cycle consistency loss specifically designed for recovery. While the generator learns to embed the adversarial signal through makeup transfer, the recovery module learns to remove it. This ensures that the transformation is mathematically reversible, allowing authorized parties who possess the recovery module to "strip" the makeup and restore the original identity.
Experimental Results
The researchers evaluated RMT-GAN using the Makeup Transfer (MT), CelebA-HQ, and LADN datasets, alongside commercial black-box APIs like Face++ and Aliyun.
Attack Performance and Visual Quality
RMT-GAN significantly outperformed state-of-the-art (SoTA) methods. On the CelebA-HQ dataset, it achieved an average Attack Success Rate (ASR) of 66.9%, compared to 44.9% for AMT-GAN and 12.5% for Adv-Makeup. In a user study with 25 participants, RMT-GAN's images were rated as superior in naturalness and quality 88% of the time.
Successful Identity Recovery
The recovery module proved highly effective in restoring utility for authorized systems. For instance, while the adversarial attack reduced recognition rates on the MobileFace model to nearly zero, the recovery module restored it to 17.2%, very close to the original source's 19.4% baseline.
Conclusion
RMT-GAN transforms adversarial attacks from destructive processes into functional privacy tools. By prioritizing visual naturalness and recoverability, it provides a robust defense against unauthorized face recognition without sacrificing the utility of photos for legitimate, authorized applications. Future work will focus on more localized makeup strategies and bridging the gap between digital transfer and physical-world application.
Summary updated