Preventing Latent Diffusion Model-Based Image Mimicry via Angle Shifting and Ensemble Learning

Minghao Li, Rui Wang, Ming Sun, Lihua JingIJCAI 2025 · DOI

Abstract

Latent Diffusion Models (LDMs) have revolutionized image generation but also enabled unauthorized mimicry of artistic styles and personal identities. Existing adversarial protection methods often fail due to the inherent denoising capabilities of LDMs or prohibitive hardware requirements. This paper presents a novel approach that utilizes angle-shifting adversarial perturbations and an alternating iterative framework to provide robust, memory-efficient protection against image mimicry.

Introduction

The rise of powerful Latent Diffusion Models (LDMs) like Stable Diffusion has introduced significant concerns regarding "unauthorized image mimicry." Malicious actors can use these models to clone an artist's unique style or a person's facial features without consent. Adversarial attacks—adding imperceptible noise to images to disrupt model processing—offer a potential defense. However, current methods face two primary challenges: the "denoising bottleneck," where the LDM's internal processes cancel out the protective noise, and high VRAM requirements that limit accessibility for typical users.

Methodology

To address these challenges, we propose three key innovations: the Angle-Shifting Attack, the Alternating Iterative Framework (AIF), and a Stable Optimization Strategy.

Angle-Shifting Attack

Traditional adversarial attacks often rely on $\ell_2$ loss to maximize error magnitude. However, LDMs are designed to remove such noise. We observe that the LDM's denoiser generates a "cancellation vector" that effectively subtracts the perturbation. By shifting the focus from magnitude to the angle of the perturbation using cosine similarity loss, we create "effective vectors" that are significantly harder for the denoiser to remove.

Visualization of the cancellation effect where the LDM's denoiser subtracts the adversarial perturbation.

Alternating Iterative Framework (AIF)

To overcome hardware constraints, the AIF mathematically decouples the optimization process. Instead of loading the entire LDM pipeline into memory, the framework alternates between calculating gradients in the latent space and applying them in the pixel space. This approach allows for high-resolution image protection on consumer-grade GPUs, such as the NVIDIA 1080Ti, without sacrificing performance.

Stable Optimization Strategy

Gradients in diffusion models can be highly stochastic due to random noise and varying timesteps. We stabilize the optimization by fixing the Gaussian noise during the attack and employing a Grouped Gradient Ensemble. This ensemble averages gradients across multiple timesteps, ensuring the attack disrupts both fine details and global structures.

Experimental Evaluation

The proposed method, CosAttack, was evaluated on the CelebA-HQ (500 faces) and WikiArt (540 images across 27 styles) datasets.

Key Results

CosAttack consistently outperforms state-of-the-art methods like AdvDM and Mist. On the CelebA-HQ dataset, CosAttack achieved an FID score of 99.46, significantly higher than AdvDM's 84.71, indicating superior protection. Qualitatively, the method induces severe texture disruption and high-frequency noise in the LDM's output, rendering the reconstructed images unrecognizable.

Example of severe texture disruption achieved by the proposed protection method.

Furthermore, the AIF framework demonstrated exceptional efficiency. While competing methods failed to process 512x512 images on a 12GB VRAM GPU, CosAttack maintained full functionality and high protection levels.

Conclusion

By leveraging angle-shifting perturbations and a memory-efficient alternating framework, this work provides a practical and robust defense against unauthorized image mimicry by LDMs. The "plug-and-play" nature of the AIF suggests it could be integrated into various existing adversarial frameworks to enhance their accessibility and effectiveness.

Summary updated

BibTeX
@inproceedings{li2025preventing,
  author = {Minghao Li and Rui Wang and Ming Sun and Lihua Jing},
  title = {{Preventing Latent Diffusion Model-Based Image Mimicry via Angle Shifting and Ensemble Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/156},
}
Figure