Diffusion models generate images by starting from random noise and denoising it step by step. Many can also be "unlearned": fine-tuned so that a prompt such as "nudity" or "Van Gogh style" no longer yields that content. Recent attacks show this forgetting is often shallow. The attack IVO leaves the prompt alone and optimizes only the starting noise (The Illusion of Forgetting). RECALL adds an optimized image prompt (Image Can Bring Your Memory Back). Both bring the erased concept back in most attempts. The best robust-erasure defenses harden only the text encoder, and they do it with an expensive inner attack loop. That loop is the min-max procedure of adversarial training, in which each training step first searches for a worst-case input. So the defense is costly, and it protects the wrong entry point.
We propose to measure, at every noise level, how far the erased prompt's denoising direction departs from a harmless anchor prompt's direction. We then penalize how sharply that gap can grow when the noisy image is nudged. Stated plainly: after erasure, the model should treat "nudity" like "a photo" not only on typical noisy images but in a whole neighborhood around them, so that no small push to the noise can reopen the difference. The penalty is estimated with a few extra forward passes and needs no attack loop. It applies only to the concept-specific part of the network's output. That targeting is what should keep general image quality intact, because the denoiser as a whole must stay sensitive to its input in order to denoise.
Core Claim
Treat the sampler as the reverse-time stochastic differential equation of Song et al.. This is the continuous-time view in which a denoising network, called the score function, steers noise back toward data. Let $s_\theta(x_t, t, c)$ be the score at noisy image $x_t$, time $t$ and prompt $c$. Define the concept residual
$$r_\theta(x_t, t) = s_\theta(x_t, t, c_{\text{forget}}) - s_\theta(x_t, t, c_{\text{anchor}}),$$
where $c_{\text{forget}}$ is the erased prompt and $c_{\text{anchor}}$ a neutral prompt. Standard erasure methods, such as ESD in Erasing Concepts from Diffusion Models, push $r_\theta$ toward zero on noisy images drawn from ordinary sampling trajectories. A latent attack works by finding starting noise whose trajectory passes through regions where $r_\theta$ is still large.
In the reverse process, the gap between a forget-prompted trajectory and an anchor-prompted trajectory from the same start grows at a rate $g(t)^2 r_\theta$, where $g(t)$ is the noise scale of the process. Its total size is bounded by the time integral of $g(t)^2 \|r_\theta\|$ along the path. It follows that bounding how fast $r_\theta$ can change near the data, at every $t$, limits how much any bounded change to the start can resurrect the concept.
We implement this with the first-order argument of A Unified Gradient Regularization Family for Adversarial Examples. That paper shows the worst-case inner maximization reduces, to first order, to an input-gradient norm penalty. We apply the argument to $r_\theta$ instead of to a classification loss. To avoid double backpropagation we use a random-probe finite difference:
$$\mathcal{L} = \mathcal{L}_{\text{erase}} + \lambda\, \mathbb{E}_{t,\,x_t,\,v}\!\left[ g(t)^2\, \frac{\| r_\theta(x_t + h v, t) - r_\theta(x_t, t)\|^2}{h^2} \right] + \mu \|\theta - \theta_0\|^2 .$$
The symbols are as follows. $\mathcal{L}_{\text{erase}}$ is the ESD erasure loss. $v$ is a random unit direction in latent space. $h$ is a small step. $\theta_0$ are the pretrained weights. $\lambda$ and $\mu$ are weights on the two penalties. The $g(t)^2$ weighting comes directly from the rate at which $r_\theta$ enters the reverse dynamics.
What is new has three parts. The robustness term acts on the image-denoising pathway that IVO and RECALL exploit, not on the text encoder. It is loop-free. And it is confined to the concept residual, rather than smoothing the whole denoiser as a contractive penalty would. Such a contractive penalty, like the one in Towards Deep Neural Network Architectures Robust to Adversarial Examples, would damage generation.
Research Program
The method is built on Stable Diffusion v1.4, whose training corresponds to a discretized variance-preserving stochastic differential equation. This lets us run the same weights under a stochastic sampler and under the deterministic probability-flow ODE sampler. We update only the UNet cross-attention projections, the layers where Continual Unlearning for Text-to-Image Diffusion Models locates most semantic interference.
This choice fits the method closely. The residual $r_\theta$ is zero whenever the two prompts coincide, so its dependence on the prompt runs through cross-attention. Cross-attention is therefore the natural handle for reshaping the residual, and restricting updates to it keeps drift from the pretrained model small.
Each training step costs one ESD step plus two extra UNet forward passes: the probed pair for one random direction, with the anchor evaluation shared. We estimate the total at under twice plain ESD. For comparison, AdvUnlearn runs multi-step prompt attacks inside every step.
The analysis has three parts. First, we will check empirically whether the penalty reduces the maximum residual magnitude that a PGD attack on $x_T$ can reach, and whether that maximum predicts attack success. This tests the bound argument directly. Second, we will measure how the penalty's effect depends on noise level by switching it off in high-noise and low-noise bands. Third, we will study how the fixed random probes compare with a single power-iteration probe. That probe costs one backward pass but targets the steepest direction.
Baselines
The naive reference points are two. One is the original SD v1.4. The other is ESD alone, the same erasure loss without the sensitivity term. ESD with our parameter penalty is the exact ablation that isolates the proposal's key ingredient, since everything except the residual penalty is held fixed. ESD's reported vulnerability is large: AdvUnlearn reports roughly 73.24% attack success on nudity, and S-GRACE reports 80.00% under UnlearnDiffAtk.
The strongest existing method on prompt attacks is AEGIS. It combines an adversarially optimized erasure target, the approximate semantic center of the concept, with retention-data-free gradient projection. It reports 8.45% attack success on nudity under UnlearnDiffAtk and a COCO-10k FID of 17.43, against 64.79% and 19.34 for AdvUnlearn in the same table.
AdvUnlearn itself is the strongest method among those that pay for an adversarial inner loop. It reports 21.13% attack success in its own evaluation and a COCO-30k FID of 19.34, against 16.70 for vanilla SD.
S-GRACE is the strongest on cost. It reports erasing nudity in about four minutes, compared with over an hour for the baselines it tests. We found no reported result on Lacuna for AEGIS, AdvUnlearn or S-GRACE under IVO or RECALL. Producing those numbers is part of this work.
All methods will be retrained from released code on the same SD v1.4 checkpoint, the same concept lists and the same GPU type. Each method gets an equal hyperparameter search budget, which we fix at the same number of full training runs, selected on a held-out prompt split. Wall-clock time will be reported for every method.
Beating the field means three things at once. On IVO and RECALL, attack success must be clearly below AEGIS and AdvUnlearn. On UnlearnDiffAtk, attack success must be within a few points of AEGIS's 8.45%. COCO FID must stay at or below AEGIS's 17.43. Training time must also be comparable to S-GRACE rather than to AdvUnlearn.
Why not just…?
Why not just sample with the stochastic sampler and let the injected noise act as randomized smoothing? The stochastic sampler adds fresh noise at every step. The Blessing of Randomness shows this stabilizes editing, and it resembles the randomized smoothing defenses of Certifying Confidence via Randomized Smoothing. But smoothing only certifies the prediction averaged over the noise. An attacker who optimizes the expected outcome over sampler noise can still target a residual that is large on average. We will test this directly by running IVO under ODE sampling, SDE sampling and SDE sampling with an attacker that averages over the noise, on every erased model. If stochastic sampling alone drives attack success down against the averaging attacker, the regularizer is unnecessary and we will say so.
Why not just run latent-space adversarial training on the UNet, with a single-step attack to keep it cheap? This is the real threat. One-step latent adversarial training is close to a first-order method. It could match our penalty at a similar cost of one extra backward pass. We include it as a baseline: an FGSM-style perturbation of $x_t$ that maximizes $\|r_\theta\|$, with the same target layers and budget. We also include a five-step PGD version as the higher-cost bound. If one-step adversarial training matches us, our contribution narrows to the residual targeting and the $g(t)^2$ time weighting. We will ablate both by running the adversarial baseline on the full score rather than on the residual, with and without time weighting.
Why not just combine AEGIS or AdvUnlearn with a stronger weight penalty? Weight penalties and text-encoder hardening restrict what the model can change. They say nothing about how the unchanged denoising pathway behaves on unusual latents. That pathway is exactly the one IVO and RECALL exploit, reporting about 80% and 73.4–97.4% attack success respectively. Our table of these methods under latent attacks will show whether the gap exists in practice.
Why not just filter outputs with a safety classifier? Output filters are reactive and can be removed from open weights. As Towards Deep Neural Network Architectures Robust to Adversarial Examples notes, stacking a filter onto a model gives an attacker a single differentiable pipeline to attack end to end. We do not treat filtering as a substitute for changing the weights.
Proposed Experiments
Erasure tasks. We erase nudity and evaluate on I2P, and on the Nude-118 and NSFW-High prompt sets used by IVO. We erase Church and Parachute on Imagenette, scored by classifier accuracy on the erased class and on the other classes. We erase styles on UnlearnCanvas, using unlearning accuracy and in-domain and cross-domain retain accuracy.
Utility. We report FID and CLIP score on COCO-10k captions.
Attacks. We run UnlearnDiffAtk on prompts, IVO on the starting noise and RECALL on image prompts. Each attack uses its published budget, plus a doubled budget to check that robustness gains do not vanish under a stronger attacker.
Decisive check 1. Is latent-attack success under IVO and RECALL substantially lower than ESD's, while COCO FID stays near the ESD-with-weight-penalty ablation? If FID degrades toward the values above 100 that AdvUnlearn reports for other robust methods, the residual targeting has failed.
Decisive check 2. The same comparison against one-step latent adversarial training at matched wall-clock time. This decides whether the analytical penalty is more than a relabeled cheap attack.
Decisive check 3. Sequential erasure of up to ten UnlearnCanvas styles. Continual Unlearning for Text-to-Image Diffusion Models reports that baselines collapse after 3–6 requests and that its gradient projection keeps nearly 90% accuracy on related styles. This tests whether our method's added smoothing slows or speeds that collapse, with and without their projection layered on top.
References
- The Illusion of Forgetting: Attack Unlearned Diffusion via Initial Latent Variable Optimization
- Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearning
- Score-Based Generative Modeling through Stochastic Differential Equations
- Erasing Concepts from Diffusion Models
- A Unified Gradient Regularization Family for Adversarial Examples
- Towards Deep Neural Network Architectures Robust to Adversarial Examples
- Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspective
- Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models
- Rethinking Robust Adversarial Concept Erasure in Diffusion Models
- AEGIS: Adversarial Target-Guided Retention-Data-Free Robust Concept Erasure from Diffusion Models
- The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing
- Certifying Confidence via Randomized Smoothing