Abstract
Latent Diffusion Models (LDMs) have revolutionized image generation but also enabled unauthorized mimicry of artistic styles and personal identities. Existing adversarial protection methods often fail due to the inherent denoising capabilities of LDMs or prohibitive hardware requirements. This paper presents a novel approach that utilizes angle-shifting adversarial perturbations and an alternating iterative framework to provide robust, memory-efficient protection against image mimicry.
Introduction
The rise of powerful Latent Diffusion Models (LDMs) like Stable Diffusion has introduced significant concerns regarding "unauthorized image mimicry." Malicious actors can use these models to clone an artist's unique style or a person's facial features without consent. Adversarial attacks—adding imperceptible noise to images to disrupt model processing—offer a potential defense. However, current methods face two primary challenges: the "denoising bottleneck," where the LDM's internal processes cancel out the protective noise, and high VRAM requirements that limit accessibility for typical users.
Methodology
To address these challenges, we propose three key innovations: the Angle-Shifting Attack, the Alternating Iterative Framework (AIF), and a Stable Optimization Strategy.
Angle-Shifting Attack
Traditional adversarial attacks often rely on $\ell_2$ loss to maximize error magnitude. However, LDMs are designed to remove such noise. We observe that the LDM's denoiser generates a "cancellation vector" that effectively subtracts the perturbation. By shifting the focus from magnitude to the angle of the perturbation using cosine similarity loss, we create "effective vectors" that are significantly harder for the denoiser to remove.
Alternating Iterative Framework (AIF)
To overcome hardware constraints, the AIF mathematically decouples the optimization process. Instead of loading the entire LDM pipeline into memory, the framework alternates between calculating gradients in the latent space and applying them in the pixel space. This approach allows for high-resolution image protection on consumer-grade GPUs, such as the NVIDIA 1080Ti, without sacrificing performance.
Stable Optimization Strategy
Gradients in diffusion models can be highly stochastic due to random noise and varying timesteps. We stabilize the optimization by fixing the Gaussian noise during the attack and employing a Grouped Gradient Ensemble. This ensemble averages gradients across multiple timesteps, ensuring the attack disrupts both fine details and global structures.
Experimental Evaluation
The proposed method, CosAttack, was evaluated on the CelebA-HQ (500 faces) and WikiArt (540 images across 27 styles) datasets.
Key Results
CosAttack consistently outperforms state-of-the-art methods like AdvDM and Mist. On the CelebA-HQ dataset, CosAttack achieved an FID score of 99.46, significantly higher than AdvDM's 84.71, indicating superior protection. Qualitatively, the method induces severe texture disruption and high-frequency noise in the LDM's output, rendering the reconstructed images unrecognizable.
Furthermore, the AIF framework demonstrated exceptional efficiency. While competing methods failed to process 512x512 images on a 12GB VRAM GPU, CosAttack maintained full functionality and high protection levels.
Conclusion
By leveraging angle-shifting perturbations and a memory-efficient alternating framework, this work provides a practical and robust defense against unauthorized image mimicry by LDMs. The "plug-and-play" nature of the AIF suggests it could be integrated into various existing adversarial frameworks to enhance their accessibility and effectiveness.
Summary updated