Introduction
The rapid advancement of text-to-image (T2I) diffusion models has enabled sophisticated personalization techniques like DreamBooth and Textual Inversion. These methods allow users to generate high-fidelity images of specific individuals using only a small set of reference photos. However, this capability poses significant risks, including the creation of non-consensual deepfakes, the spread of misinformation, and the violation of individual privacy.
The Challenge of Anti-Personalization
Existing "anti-personalization" strategies typically rely on adversarial attacks to "poison" training data. The goal is to ensure that any model fine-tuned on these protected images produces distorted or unrecognizable outputs. However, current methods often suffer from a critical trade-off: they either fail to provide robust protection against sophisticated fine-tuning or introduce visible artifacts that degrade the original image's aesthetic quality, making them unsuitable for social media sharing.
The DDAP Framework
To address these issues, the authors introduce Dual-Domain Anti-Personalization (DDAP). This framework employs a multi-pronged strategy to disrupt the personalization process by attacking both the spatial and frequency domains:
- Spatial Perturbation Learning (SPL): Unlike previous methods that target the U-Net's denoising process, SPL focuses on the Variational Autoencoder (VAE) encoder. Since the VAE is typically frozen during the personalization phase, it provides a stable and effective target for adversarial perturbations, leading to structural distortions in the generated output.
- Frequency Perturbation Learning (FPL): Diffusion models are particularly sensitive to high-frequency information for achieving photorealism. FPL utilizes the Discrete Cosine Transform (DCT) to embed adversarial noise within specific frequency bands. This disrupts the model's ability to capture fine facial contours while remaining nearly invisible to the human eye.
- Localization Module: To maximize stealth, DDAP uses Diffusion Attentive Attribution Maps (DAAM) to identify the specific pixels the model focuses on when learning a new concept (e.g., a face). Perturbations are applied only to these critical regions, preserving the background and overall image integrity.
Experimental Evaluation
The researchers evaluated DDAP using the VGGFace2 and CelebA-HQ datasets, targeting Stable Diffusion v2.1. DDAP was compared against leading baselines, including Anti-DB, PhotoGuard, and AdvDM.
Key Results
DDAP demonstrated superior performance in both defense effectiveness and image stealthiness:
- Defense Effectiveness: On the VGGFace2 dataset, DDAP achieved a Face Detection Failure Rate (FDFR) of 0.90, significantly outperforming the previous state-of-the-art method, Anti-DB (0.76). This indicates that 90% of the generated "fake" faces were too distorted to be recognized by detection algorithms.
- Identity Protection: The Identity Score Matching (ISM)—a measure of how closely the generated image resembles the target—dropped to 0.15, confirming that the model failed to learn the individual's features.
- Visual Fidelity: DDAP maintained a high PSNR (31.93) and a low LPIPS (0.25), indicating that the protected images remain visually indistinguishable from the originals to the human eye.
Limitations and Future Work
While DDAP represents a significant step forward, the authors acknowledge several areas for improvement:
- Black-Box Robustness: The effectiveness of the defense can decrease when there is a mismatch between the model used for protection (e.g., SD v2.1) and the model used by an attacker (e.g., SD v1.5).
- Transmission Loss: The impact of image compression and processing pipelines common on social media platforms (like Instagram or X) remains a challenge, as these processes can strip away the high-frequency perturbations used by FPL.
Conclusion
DDAP offers a robust, proactive defense against unauthorized AI personalization. By targeting the "frozen" vulnerabilities of the VAE and exploiting the frequency sensitivity of diffusion models, it provides a high level of protection while maintaining the visual integrity of the user's original photos.
Summary updated