Do Erased Styles Come Back When You Hand the Model Their Color Statistics?

When an artist opts out of a text-to-image model, the usual remedy is "concept unlearning": edit the model's weights so that prompts naming the artist no longer produce their style. Evidence is building that this often hides the style rather than removing it. The style can be recovered if generation starts from the right intermediate state, and it can return after unrelated fine-tuning. A second, separate body of work on diffusion models (generators that turn random noise into an image by removing noise step by step) shows that much of an image's global style is carried by very simple numbers: the per-channel mean and standard deviation of the model's internal compressed image representation, called the latent. Matching only those two statistics in the starting noise gives a model a large head start toward a target style.

Put these together and you get a cheap attack, and also a cheap test that creators can run themselves. Take a few of the artist's own images, compute their latent statistics, start an "unlearned" model from noise with those statistics, and check whether the erased style comes back. This note proposes to measure that effect carefully against a model that never learned the artist. It then uses a ladder of probes, built from adaptive normalization operations, to find out which part of the latent carries the hidden style: the low-order statistics or the spatial structure. Finally, it tests whether unlearning trained against those probes actually closes the gap.

Core Claim

Weight-space unlearning is mostly evaluated by changing the prompt. We claim it should be evaluated by changing the starting latent, and that the correct comparison point is a model that never saw the artist, not the original model. Our hypothesis is that unlearned models respond much more strongly to an artist's latent mean and variance than a never-trained model does, and that this excess explains a large share of the concealment found by the partial-diffusion attack. If the hypothesis holds, two things follow. Creators get a verification test that needs only their own images and the ability to set the starting latent, which an image-to-image interface already provides. Unlearning methods also get a concrete training target. If it fails, because hidden style lives in spatial structure that survives whitening, that is a useful negative result. It would rule out the inference-time "normalize the latents back to generic" defenses the literature has proposed but never tested.

Prior Work and Where This Sits

The motivating evidence comes from two critiques of unlearning. Unlearning or Concealment? shows that ESD- and AC-unlearned models reconstruct Van Gogh or Greg Rutkowski when the original model runs the first $\psi \approx 0.55$ fraction of denoising. It introduces concept confidence and concept retrieval scores, but it offers no defense and no account of what in the handed-over latent carries the style. Its attack also requires the original weights. Unstable Unlearning shows that erased concepts resurge after benign fine-tuning. We reuse both protocols. We differ in replacing the original-model seed with seeds anyone can compute, and in measuring leakage relative to a never-trained reference.

The mechanism comes from adaptive latent normalization. Diffusion in Style samples starting noise from the element-wise mean and standard deviation of a style set's latents and adapts Stable Diffusion within 1,000 iterations, in under 5 minutes for a 3-image set. Style Injection in Diffusion applies AdaIN (shifting a feature map's mean and variance to match a reference) to the starting latent to fix the color palette. TALE and DiffMorpher apply the same operation to intermediate latents. These works treat moment matching as a tool for imposing style. We use it as a measuring instrument: it lets us give a model a style's statistics without its structure, or its structure without its statistics. These papers also constrain us, because they show that moments mainly carry color and tone, while brushwork needs attention swapping or Gram-matrix terms. We therefore expect moment probes to revive only part of a style, and the analysis is built to quantify that fraction rather than assume it is large.

The unlearning methods under test are ESD/AC (as evaluated by Sharma et al.) and PURE. PURE projects cross-attention key/value weights away from a forget subspace in closed form, and on the Holistic Unlearning Benchmark (HUB) it reduces the "Pikachu" target proportion to 0.077, against 0.463 for CURE. PURE was evaluated against paraphrased and adversarial prompts, not against modified starting latents. ReTrack redirects denoising trajectories toward nearest retain neighbors through fine-tuning. It is the closest existing method in spirit, but it targets individual data points, not styles, and was not tested against partial diffusion. The nearest intersection paper, Latent Diffusion Unlearning, is a different problem: it perturbs an artist's images before release so that personalization fails. We study what remains inside a model after the model owner has tried to honor an opt-out.

Finally, GenAI Copyright Evidence with Operational Meaning argues that copyright evidence should be framed as a hypothesis test with a controlled false-positive rate. We adopt that framing for the creator-side test.

Research Program

The central design problem is a confound. Starting a model from an artist's statistics, or from a noised version of their painting, pushes any model toward that artist's look, including a model that never saw them. Leakage is only meaningful as excess over that baseline. We therefore build a controlled setting in which a true never-trained reference exists.

We select about 24 artists from the AICD reference corpus in From Imitation to Innovation. We draw from its children's-picture-book and contemporary sub-corpora and keep only artists that a style classifier, trained on that corpus, does not detect in base Stable Diffusion outputs. We fine-tune Stable Diffusion on 12 of these artists to get model $F$. The base model before fine-tuning is the reference $R_0$, which has never been trained on any of them. We then unlearn 6 artists from $F$ with each method to get $U$, and keep the other 6 as retain artists. Real-world spot checks repeat the analysis on the HUB style concepts (Van Gogh, Picasso, Frida Kahlo), where no never-trained reference exists and results are reported relative to the original model only.

Each probe $p$ is applied to all three models. We measure style retrieval $S$, defined as the classifier's probability for the target artist, plus the concept retrieval score of Sharma et al. We then report a normalized excess:

$$E(p) = \frac{S_U(p) - S_{R_0}(p)}{S_F(p) - S_{R_0}(p)}$$

Here $S_U$, $S_F$ and $S_{R_0}$ are the mean retrieval for the unlearned, fine-tuned and never-trained models under the same probe. $E=0$ means the unlearned model is indistinguishable from one that never learned the artist, and $E=1$ means unlearning did nothing under that probe.

The probes form a ladder. All are built from held-out artist images that were never in the fine-tuning set, so memorization cannot drive the result. The bottom rung is ordinary noise with paraphrased style prompts, which is the standard evaluation. The next rung is moment-seeded noise: starting latents drawn from the channel-wise, and separately the element-wise, mean and variance of 3 to 10 of the artist's encoded images, following Diffusion in Style. Above that is whitened structure: a held-out artist image noised to step $t$ and then AdaIN-normalized to generic latent moments, which keeps layout and texture but removes palette. The next rung is the full noised latent at the same $t$, equivalent to image-to-image editing. The top rung is the Sharma et al. partial-diffusion attack with $F$ as the seeding model. Every rung uses a sweep over start step $t$ (or $\psi$), and the analysis reports how much of the top-rung excess the moment-only and structure-only rungs each recover.

The constructive part is moment-aware unlearning. During ESD-style unlearning, the forget-set training trajectories are seeded from moment-matched noise and from whitened partially noised latents, not only from standard noise. The training target is unchanged. Two ablations isolate the ingredient: training with moment seeds only, and training with structure seeds only.

The creator-facing output is a calibrated test. Using the 12 never-fine-tuned artists, we set a threshold on $S_U - S_{R_0}$-style statistics, estimated without access to $F$, so that the false-positive rate is at most a chosen level. We then report the detection rate for artists that were truly fine-tuned in and then unlearned.

Baselines

The naive reference point is the standard evaluation: prompt-only leakage under paraphrase, which is how unlearning is usually declared successful. If moment-seeded probes show no more excess than paraphrases, the proposal's premise is wrong.

The strongest existing unlearning method is PURE. It is closed-form, takes about a minute, and its reported HUB result (Pikachu target proportion 0.077 versus 0.463 for CURE) is the best number in the file. The file reports no number for PURE on the HUB artistic-style category, and no reported CCS value at $\psi \approx 0.55$ from Sharma et al. We will run PURE with its published retain-anchor procedure, on the same fine-tuned model and the same unlearned artists, with an equal anchor budget. ESD is included because Sharma et al. use it to demonstrate concealment.

The strongest defense competitor, and the one that most directly threatens the proposal, is ESD trained against partial-diffusion seeds from $F$. It gets the same number of unlearning steps and the same retain regularization as moment-aware unlearning. Moment-aware unlearning counts as beating it if it reaches lower $E$ on the moment and full-latent rungs at matched retain quality. Retain quality is measured by MS-COCO FID and by retain-artist style accuracy within a fixed tolerance. It must also show lower resurgence after the benign fine-tuning protocol of Unstable Unlearning.

Two further reference points complete the comparison. The first is an inference-time normalization defense that re-normalizes intermediate latents to generic moments, the untested idea from the literature. The second is the moment-only and structure-only training ablations described above.

Why not just…?

Why not just use the partial-diffusion attack as the evaluation? It requires the original weights, which a creator auditing a hosted or retrained model will not have. It also bundles statistics and structure together, so it cannot say what a defense must remove. We keep it as the top rung, and the decomposition tells us whether a probe that needs no special access recovers most of its effect.

Why not just run image-to-image from the artist's own paintings? That is a real and simpler probe, and we include it as the full-latent rung. Without the never-trained reference, though, it cannot be interpreted: a sufficiently noised painting yields the artist's look from any model. The excess measure $E$ is what turns it into evidence. If image-to-image alone carries all the excess and moment seeds add nothing, we will say so and drop the moment-seeding claim.

Why not just normalize latents back to generic statistics at inference time? It can only be enforced by whoever controls sampling, so it does nothing for released weights. Moment matching also controls color, so blanket normalization should damage legitimate generation. Diffusion in Style and TALE both report sensitivity to how long normalization is applied. We test it directly as a baseline. If it lowers $E$ at acceptable FID, it is a legitimate cheap option for hosted services, and we will report it that way.

Why not just add adversarial seeds to unlearning training, without any moment analysis? This is the strongest threat. Moment-aware unlearning is a specific instance of it, and the partial-diffusion-seeded ESD baseline is the general version. The added value we claim is that moment seeds are cheap, need no original weights, and may cover the leakage that matters. If the general version wins at matched cost, the contribution reduces to the evaluation protocol and the decomposition, and we will report it that way.

Proposed Experiments and Decisive Checks

The first decisive check is whether moment-seeded probes give $E$ well above the paraphrase rung for ESD and PURE in the controlled setting, with confidence intervals across artists and seeds. If they do not, the moment hypothesis is rejected. The second check is the decomposition: what fraction of partial-diffusion excess is recovered by the moment-only rung versus the whitened-structure rung. That fraction decides whether normalization-based defenses are worth building at all. The third check is whether moment-aware unlearning reduces $E$ across all rungs, including the partial-diffusion rung it was not trained on. It must do so without exceeding the FID and retain-accuracy tolerance, and it must hold after benign fine-tuning. The fourth check is whether the calibrated creator test reaches a useful detection rate at a 5% false-positive rate using only 3 to 10 of the artist's images. The HUB spot checks then show whether the controlled findings transfer to famous styles, reported without the never-trained normalization and clearly labeled as such.