Artists who publish work online face a specific threat: a stranger can download a handful of their images and, with a few minutes of fine-tuning on a text-to-image diffusion model — a generator that learns by gradually denoising random noise into images — produce unlimited imitations of their style. The main countermeasure is proactive poisoning: add imperceptible noise to published images so that fine-tuning on them fails. But the field has settled into a bad equilibrium on cost, from both sides. Generating protective noise takes minutes per image of bi-level optimization, which is impractical for an artist with a portfolio; meanwhile an imitator can defeat the protection almost for free, because CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models shows that training small low-rank adapter modules on just the autoencoder — the component that compresses images into the diffusion model's internal representation — restores mimicry against nine published defenses.
This proposal argues the defender should stop trying to make corruption hard to remove and instead make it costly to remove. The key observation, drawn from CAT and Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models (FastProtect), is that today's protective noise corrupts the autoencoder in directions that real images never use — which is exactly why a cheap adapter can learn to undo it without touching anything the imitator cares about. We propose placing the corruption inside the "signal subspace": the directions along which clean images genuinely vary. Any correction trained to restore mimicry must then also distort legitimate images, forcing the attacker into the classic robustness-accuracy trade-off — degraded clean fidelity as the price of stolen style. The same design choice, we claim, makes the protection cheaper to generate and more likely to transfer across diffusion models with different autoencoders, because natural-image statistics are shared across architectures in a way any single encoder's quirks are not.
Core Claim
Three falsifiable claims. First, a perturbation whose induced autoencoder error lies in the principal subspace of clean latent variation cannot be removed by a reconstruction-trained adapter without a measurable fidelity loss on benign images; we will demonstrate this as an explicit attacker-side trade-off curve (mimicry recovery versus clean-image degradation) that existing defenses do not produce. Second, because the target subspace is a property of natural images rather than of one encoder, subspace-aligned perturbations transfer across structurally distinct backbones better than encoder-specific attacks such as Variance as a Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models, which maximizes latent variance for one specific variational autoencoder. Third, the subspace is estimated once from public clean data, so per-image protection reduces to short projected-gradient steps against a surrogate with a fixed regularizer, or a precomputed bank lookup in the spirit of FastProtect, without inheriting FastProtect's dependence on one autoencoder's latent clusters.
Research Program
The mechanism is one objective. Let $E$ be the surrogate autoencoder encoder, let $e = E(x+\delta) - E(x)$ be the latent error caused by perturbing image $x$ by $\delta$, and let $\Sigma$ be the covariance matrix of encoder latents over a large corpus of clean, unprotected images (estimated once). We optimize
$$ \max_{\|\delta\|\le\epsilon} \; L_{\mathrm{disrupt}}(x+\delta) \;+\; \lambda \, \frac{e^\top \Sigma \, e}{e^\top e}, $$
where $L_{\mathrm{disrupt}}$ is any existing disruption loss — we will use the diffusion denoising-loss maximization of Unlearnable Examples for Diffusion Models: Protect Data from Unauthorized Exploitation — and the second term is a Rayleigh quotient: the fraction of the error's energy that falls in the high-variance directions of clean latents. The intuition is an identifiability argument. An adapter trained by reconstruction loss on protected images must decide which latent components are "damage" to remove and which are "content" to keep. If the damage is orthogonal to everything clean images ever do, that decision is trivial. If the damage is statistically indistinguishable from content, the adapter faces an estimation problem it cannot win: removing the corruption removes real signal. This is the generative analogue of the lesson in Transferable Unlearnable Examples, where unlearnable noise transfers only when it targets task-intrinsic structure (class separability) rather than model-specific shortcuts; we target natural-image latent structure instead, which also removes their need for class labels that artistic corpora do not have.
We will build three components: a one-time subspace estimator over a public clean corpus (top-$k$ eigenvectors of $\Sigma$, with $k$ a studied hyperparameter); the perturbation optimizer above with only $\lambda$ and $k$ as new knobs; and a bank variant that stores subspace-aligned perturbation templates indexed like FastProtect's mixture-of-perturbations, for artists who need near-zero per-image cost. The threat model follows CAT: the imitator has consumer-grade compute, may attach low-rank adapters to the autoencoder, and may possess clean side data from other artists to regularize training — the hardest case, since side data makes the trade-off, not the alignment failure, the only line of defense. We deliberately do not assume the attacker runs full min-max adversarial training; the file's evidence (Bypassing the Computational Bottleneck of Adversarial Training; Theoretically Principled Trade-off between Robustness and Accuracy) is that its cost and fidelity penalty make it uneconomical, which is precisely the asymmetry we exploit. Our design also speaks to the open contradiction in the literature — whether protection lives in the autoencoder or in the U-Net denoiser — by taking CAT's autoencoder evidence seriously and weaponizing it, while Anti-DreamBooth's U-Net-side view serves as the competing ablation.
Baselines
The "do nothing" reference is publishing unprotected images and letting the attacker run plain DreamBooth-style fine-tuning with low-rank adapters on Stable Diffusion, as introduced by DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. CAT describes its adapter-corrected result on protected data, a Face Similarity Score of 0.572, as nearly identical to training on unprotected data, so roughly 0.57 is the unprotected reference point; any defense must drive similarity far below that under attack. The strongest competitor is the strongest existing defense evaluated under the strongest existing attack: Mist, Glaze (GLAZE: Protecting Artists from Style Mimicry by Text-to-Image Models), and Anti-DreamBooth, each run at its published perturbation budget and then attacked with CAT's adapter correction under identical adapter rank, training steps, and data. The number to beat is CAT's reported restoration of Face Similarity Score from 0.160 to 0.572 on Mist-protected images; success means holding similarity at or below about 0.2 after the same attack, at perturbation stealth within the range that Evaluating Adversarial Protections for Diffusion Personalization: A Comprehensive Study documents as acceptable. StyleGuard is the competitor for robustness-by-simulating-the-attacker; it will be run with the same per-image wall-clock budget as our optimizer, and tested against VAE-adapter attacks it has never been evaluated on — no reported result found on Lacuna for StyleGuard under VAE-adapter attacks, nor a per-image latency figure. FastProtect is the efficiency competitor at a reported 0.04 seconds per 512-pixel image on an A100, versus 225 seconds for Anti-DreamBooth; we must stay within an order of magnitude of FastProtect, and we will run its bank under CAT's attack, for which no reported result found on Lacuna. Finally, the key ablation is the identical optimizer with the alignment term switched off ($\lambda = 0$) at matched perturbation budget and compute; this isolates subspace alignment as the ingredient responsible for the attacker-side fidelity cost, since without it the perturbation is simply another encoder-targeted attack in the family that CAT already breaks.
Why not just…?
Why not just crank up the perturbation budget? Larger noise is the cheapest possible fix, but it runs into the stealth-versus-disruption trade-off that Evaluating Adversarial Protections documents explicitly across eight methods, and amplitude does not change where the corruption lives: Glaze already reports over 85% efficacy against Gaussian noise and JPEG compression at a budget of 0.05, yet CAT still neutralizes it, because an adapter regresses out large errors in unused directions as easily as small ones. We will nonetheless run an epsilon-sweep under the CAT attack, and if raw amplitude alone holds similarity below 0.2 at perceptually acceptable PSNR, the proposal's core claim is wrong and we will report that.
Why not just simulate the adapter inside the optimization loop? This is the most serious alternative, because StyleGuard already performs bi-level look-ahead optimization that simulates attacker fine-tuning and ensembles substitute purifiers. Three things argue against it as the primary design: it is exactly the multi-model bi-level cost that makes protection minutes-per-image and impractical for artists; it has never been evaluated against adapter attacks, so its robustness there is unknown; and look-ahead against a specific attacker tends to overfit that attacker, whereas alignment targets the geometry that any reconstruction-trained correction must share. Still, we will run a look-ahead variant at matched compute as a head-to-head, and if it matches our adapter robustness at acceptable cost, that result stands on its own and we will say so.
Why not just purify the images, or use watermarks instead? On the attacker side, off-the-shelf purification is the obvious bypass: IMPRESS restores style-mimicry accuracy on Glaze-protected art from about 42% back to 87% — but at a reported 141 seconds per image, prohibitive for scraping pipelines, and purification itself carries a clean-fidelity risk since When Does Diffusion Purification Amplify Perturbations? shows trajectory discrepancies amplify non-monotonically in early denoising steps. We will evaluate our defense under IMPRESS as a secondary attack and report it honestly, since our design does not specifically target purifiers (StyleGuard's upscale loss does). On watermarking, Attention to Neural Plagiarism: Diffusion Models Can Plagiarize Your Copyrighted Images! shows diffusion models can replicate watermarked copyrighted images, and post-hoc auditing tools such as ArtistAuditor detect theft only after the model is trained and distributed — neither prevents the harm, so they complement rather than replace proactive poisoning.
Proposed Experiments and Decisive Checks
The main experiment runs on WikiArt (styles, including the Claude Monet partition used by CAT and IMPRESS) and CelebA-HQ plus VGGFace2 (identities, the benchmarks of Anti-DreamBooth). We protect images with our method and each baseline, then attack with low-rank-adapter personalization, with and without CAT-style autoencoder adapters, including the hardest setting where the attacker mixes in clean side images. Metrics: Face Similarity Score and face-detection failure for identities (Anti-DreamBooth reports raising failure from 0.07 to 0.63 on unprotected data), FID and CLIP-based style similarity for art, and perceptual stealth via PSNR, SSIM, and LPIPS. The decisive quantity is new: the attacker-side trade-off curve — mimicry recovery on protected images versus reconstruction fidelity on held-out clean images (PSNR, LPIPS) and generation quality on unrelated styles (FID) after adapter training. Existing defenses should trace a flat curve (full recovery, no cost, as in CAT's 0.160-to-0.572 jump); our claim is a steep curve where meaningful recovery costs measurable fidelity. The second experiment tests transfer across mismatched latent spaces, protecting on a Stable Diffusion v1.5 surrogate and evaluating on structurally different backbones, against Variance as a Catalyst and the FastProtect bank — the gap that DUAW: Data-free Universal Adversarial Watermark against Stable Diffusion Customization and other encoder-targeted methods leave open. The third experiment measures cost on an A100 against FastProtect's 0.04 seconds and Anti-DreamBooth's 225 seconds, and runs the mixed-corpus study of Unlearnable Examples for Diffusion Models, which found protected-class FID of 58.17 against 7.81 for clean classes, to ask whether subspace-aligned protection damages a model's benign generative capacity when protected and clean art are fine-tuned together. The falsifiable predictions, in order: alignment holds post-adapter similarity near 0.2 where baselines recover to 0.57; recovery attempts cost at least a stated margin in clean-image PSNR; the $\lambda = 0$ ablation loses both effects; transfer to a foreign backbone beats encoder-specific attacks; and per-image cost stays within an order of magnitude of FastProtect. If the first prediction fails under the side-data attack, the honest conclusion is that cheap adapter correction is structurally unbeatable at perceptual budgets, and the field should redirect toward detection and legal remedies — a finding worth publishing on its own.
References
- CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
- Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models
- Variance as a Catalyst: Efficient and Transferable Semantic Erasure Adversarial Attack for Customized Diffusion Models
- Unlearnable Examples for Diffusion Models: Protect Data from Unauthorized Exploitation
- Transferable Unlearnable Examples
- Theoretically Principled Trade-off between Robustness and Accuracy
- Anti-DreamBooth: Protecting users from personalized text-to-image synthesis
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
- Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models
- Evaluating Adversarial Protections for Diffusion Personalization: A Comprehensive Study
- StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations
- IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AI
- When Does Diffusion Purification Amplify Perturbations?
- Attention to Neural Plagiarism: Diffusion Models Can Plagiarize Your Copyrighted Images!
- ArtistAuditor: Auditing Artist Style Pirate in Text-to-Image Generation Models
- DUAW: Data-free Universal Adversarial Watermark against Stable Diffusion Customization