Fast and Strong Image Protection against Diffusion Personalization

Jiwon Park, Jihye Kang, Namhyuk AhnarXiv 2026 · arXiv

TL;DR

The Protective Diffusion Model (PDM) is a fast, "train-once" defense that prevents AI models from learning your face or artistic style from photos. By using a conditional diffusion process to generate protective perturbations in about one second, PDM provides stronger protection than existing methods while being up to 100x faster than traditional optimization-based approaches.

What Problem Does This Paper Address?

Personalized text-to-image models (like DreamBooth or LoRA) can replicate a person’s likeness or an artist's unique style using just a handful of images. While this enables creative tools, it also fuels non-consensual imagery and impersonation fraud.

The current defense is "image protection": adding subtle, invisible perturbations to photos before they are uploaded. These perturbations are designed to make AI fine-tuning fail. However, existing state-of-the-art protections are slow. They use iterative optimization (like PGD) to craft a unique perturbation for every single image from scratch, which can take anywhere from several seconds to tens of minutes per photo. This makes protecting an entire photo album computationally prohibitive for average users.

Methods

The authors introduce PDM, a conditional diffusion model (a 36M parameter U-Net) that learns to "sample" a protective perturbation rather than optimizing it pixel-by-pixel for every new image.

Figure 2. Method overview. (Left) Per-image optimization runs a separate PGD search through the frozen surrogate for every image. (Right) PDM is trained once. A PGD teacher under the TRP loss produces perturbations with a randomized step size and step count (1), the generator is distilled on them (2), and it is then post-trained with the TRP loss as a reward, evaluated after a random purification (3). At deployment PDM protects a new image by 50-step sampling, in about one second.

The core technical innovations include:

  1. Two-Stage Training:
    • Distillation: The model first learns to imitate a slow, iterative "teacher" optimizer.
  2. Reward Post-training: The model is then fine-tuned using a differentiable reward. This "lifts" the model's performance beyond its teacher.
  3. Target Residual Projection (TRP) Loss: Most protections try to maximize the attacker's loss or match a target image's latent features. The authors argue these methods inadvertently help the attacker by shrinking the "denoising residual." TRP instead pushes the residual along a specific target direction without penalizing its size, creating a much more difficult landscape for the personalization process to navigate.
  4. Purification-Aware Training: PDM is trained to be robust against common "purification" attempts (like JPEG compression or resizing) by including these transformations directly in the training loop.

Datasets

The researchers evaluated PDM on two primary domains:

  • Faces: 20 subjects from the CelebA-HQ dataset.
  • Artworks: 20 artists from the WikiArt dataset. Training was performed on a separate set of 10,000 images (5,000 from each domain) to ensure the model generalizes to subjects it has never seen before.

Key Insights and Ideas

  • Amortization is Key: By training a generator once, the cost of protection shifts from "per-image" to a one-time "upfront" training cost. This makes real-time protection feasible.
  • Diffusion for Defense: Using a diffusion process to create the perturbation (rather than a simple one-step neural network) gives the model the capacity to match the strength of iterative optimizers.
  • The "Residual" Matters: The paper highlights that the most effective way to "confuse" a diffusion model is to manipulate its denoising residual. By forcing the model to learn a specific, consistent error pattern (the TRP), the personalized model ends up producing repetitive textures instead of the intended subject.
Figure 5. Qualitative comparison on faces and artworks. The reference column shows the training images of each subject (top two rows: faces; bottom two rows: artworks), and the remaining columns show personalized generations under the corresponding protection, all from the same sampling seed within a row.

Key Results

  • Speed: PDM protects an image in ~1.04 seconds on a consumer GPU. This is roughly 4-5x faster than the cheapest per-image baselines and nearly 100x faster than the most complex ones.
  • Stronger Protection: Despite being faster, PDM outperformed nine existing baselines on every protection metric for both faces and artworks.
  • Efficiency vs. Protection: As shown in Figure 4, even with only 5 sampling steps (taking just 0.16s), PDM still provides stronger protection than any other baseline.
Figure 4. Protection against speed. Time to protect one image (log scale, MetaCloak omitted) and the protection after personal- ization (lower is better). The PDM curve varies the number of DDPM sampling steps from 5 to 50. In the shaded region, protec- tion takes under one second and is stronger than every baseline.
  • Robustness: PDM's protection survived common image processing (JPEG, resizing) better than other methods, thanks to its purification-aware training. It even showed strong "transfer" performance, protecting images against attackers using different versions of Stable Diffusion (e.g., SDXL).

Limitations and Future Work

  • Regenerative Purifiers: The authors acknowledge that "regenerative" purifiers—tools that essentially re-draw the image (like AI-based upscalers)—can still strip away most protections, including PDM's.
  • Future Directions: The authors suggest that because PDM is trained once, its reward function could be expanded to include even more diverse "surrogate" models and purifiers, potentially making the protection even more universal and harder to bypass.

Summary updated

Abstract

Personalized text-to-image diffusion models can reproduce a person from a handful of photographs. Image protection perturbs each image before publication so that fine-tuning on it fails, but it optimizes every perturbation from scratch, so its cost grows with the number of images. We present the protective diffusion model (PDM), which is trained once and samples the perturbation of a new image over many denoising steps. PDM distills an iterative teacher and is then post-trained with the target residual projection (TRP) loss as a reward, a stronger objective than existing ones. PDM outperforms nine baselines on faces and artworks and nearly matches its teacher at a tenth of the cost.
BibTeX
@misc{park2026fast,
  author = {Jiwon Park and Jihye Kang and Namhyuk Ahn},
  title = {{Fast and Strong Image Protection against Diffusion Personalization}},
  year = {2026},
  eprint = {2611.22494},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22494},
}
Figure