Most unsupervised defect and lesion detectors work by "healing" an image. A generative model trained only on normal images redraws the input as it should look, and whatever differs between the input and the redrawing is flagged as anomalous. This works until two failure modes appear. The model can be too capable and redraw the defect along with everything else, so the difference shrinks to zero and the defect is missed. Or it can be too restrictive and blur fine normal texture, so the difference lights up on perfectly good wiring or fabric. For diffusion models (generators that learn to turn random noise into images by removing noise step by step), the trade-off sits in a single knob: how much noise to add before redrawing. Too little noise and the lesion survives. Too much and the patient's own anatomy is replaced by a generic one.
This note proposes treating that knob as a symptom of a badly posed problem. Writing an image as a normal layer plus an anomaly layer, $x = n + a$, is the same underdetermined equation that reflection-removal research has studied for years: one observed value per pixel, two unknowns. That field does not resolve the ambiguity by corrupting the input. It solves for both layers jointly, with a separate prior on each. We propose to localize anomalies the same way. At test time we solve for a normal layer that a normal-only diffusion model finds plausible and an anomaly layer that is spatially sparse, under the constraint that the two layers add up exactly to the input. No synthetic defects are used, and no noise depth has to be tuned.
Core Claim
Anomaly localization in reconstruction-based detectors fails mainly because the decomposition $x = n + a$ is solved implicitly. The noise schedule and the network's capacity decide where the anomaly ends up. Solving it explicitly, as a regularized inverse problem with a learned normal prior and a hand-specified anomaly prior, should give two things. It should give localization that is much less sensitive to its one regularization weight than diffusion reconstruction is to its noise depth. It should also give a principled way to set that weight from normal validation images alone. Concretely, we solve
$$\min_{n,\,a}\; \|x - n - a\|_2^2 \;+\; \lambda \sum_{p} \|a_p\|_2 \;-\; \log p_\theta(n),$$
where $p_\theta$ is a diffusion model trained only on normal images, $a_p$ is the anomaly layer restricted to image patch $p$, and the patch-wise group norm weighted by $\lambda$ encodes the belief that anomalies occupy few patches. The anomaly map is $|a|$ itself rather than $|x - \hat{x}|$. The trivial solution $a = 0$, $n = x$, which is the "identical shortcut" in this vocabulary, is ruled out because $x$ has low likelihood under $p_\theta$ exactly where it is defective. The opposite failure, where $n$ hallucinates new normal content, is ruled out because $n$ must still account for all of $x$ outside a sparse support.
A second, separately testable ingredient is the gradient exclusion penalty of Single Image Reflection Separation With Perceptual Losses. That paper penalizes edges that appear at the same location in both separated layers, on the premise that each physical edge belongs to one layer. We expect this to suppress the thin halos that blurred normal edges leave in anomaly maps. We also expect it to hurt "missing part" anomalies: there the anomaly layer is the negative of a component the normal layer must draw, so their edges necessarily coincide. We treat this as a hypothesis to be measured per anomaly type, not as a default.
Relation to Prior Work
Three works motivate the proposal by documenting the failure it targets. Denoising Diffusion Models for Anomaly Localization in Medical Images describes the noise-depth dilemma in medical diffusion reconstruction. Adversarial Pseudo Healthy Synthesis Needs Pathology Factorization shows that cycle-consistent translation hides lesion traces inside "healed" images. The MVTec AD paper (MVTec AD - A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection) shows autoencoder subtraction reaching image AUROC 0.86 on cable while its per-region overlap is only 0.05.
Xia et al. already factor a pathological scan into a healthy image and a pathology mask. However, they recombine the two through an unconstrained learned network trained adversarially, whereas we impose an exact additive constraint with no learned recombiner.
The closest prior work supplies a contrast rather than a method. TransFusion - A Transparency-Based Diffusion Model for Anomaly Detection already predicts normal appearance, anomaly appearance and a mask jointly inside a diffusion process. DRÆM - A discriminatively trained reconstruction embedding for surface anomaly detection trains a restorer and a segmenter on composited defects. Both learn the separation from synthetic anomalies, Perlin-noise masks filled with external textures. The anomaly layer they can recover is therefore whatever that generator can produce. Our separation is solved per image at test time, and the only learned component sees normal data exclusively. If synthetic defects match real ones, we should expect to lose to these methods. Our bet is on settings where they do not match, such as brain lesions, or where the defect is a missing or displaced part.
From layer separation we borrow two things. The first is the general stance that the ambiguity must be broken by per-layer priors, set out in Benchmarking Single-Image Reflection Removal Algorithms and Single Image Layer Separation Using Relative Smoothness. The second is the exclusion loss. We do not borrow the unpaired two-domain training of Separate in Latent Space: Unsupervised Single Image Layer Separation, because it needs example collections of both layers, and in anomaly detection the second layer is by definition unseen. Robust Subspace Recovery Layer for Unsupervised Anomaly Detection is the conceptual ancestor: it separates inliers from outliers with a robust norm, but over whole samples in a linear latent subspace. We move the robust split into pixel space and replace the linear subspace with a diffusion prior.
Research Program
Diffusion model. We train one diffusion model per category on normal images only, using the same backbone family the diffusion baselines use.
Inference. Inference alternates two steps for a small, fixed number of rounds. In the first step, $a$ is held fixed and a guided reverse diffusion pass is run on $x - a$, pulling the sample toward consistency with $x - a$. This yields a normal layer that is both plausible and faithful outside the anomaly support. In the second step, $n$ is held fixed and $a$ is updated in closed form by patch-wise shrinkage of the residual $x - n$, which is the proximal step for the group norm. The exclusion variant adds a few gradient steps on the penalty from Zhang et al.
Setting $\lambda$. We choose $\lambda$ so that, on held-out normal images, a fixed small fraction of patches carry nonzero anomaly energy. This turns the open "threshold from normal data only" problem raised in the MVTec AD paper into a false-alarm budget that can be stated in advance.
Sensitivity analysis. The main analysis compares sensitivity curves. For the strongest diffusion reconstruction baseline we sweep the noise depth across its full range, and for our method we sweep $\lambda$ across a comparable range. Both are reported as localization quality against setting, so the claim of reduced sensitivity is shown directly rather than asserted. We will also report wall-clock time honestly. Alternating optimization is slower than a single pass, and TransFusion already costs 0.34 s per image against DRÆM's 0.05 s by its own report. Speed is not a claim of this work.
Baselines
Naive subtraction. The reference point is plain reconstruction subtraction with a convolutional autoencoder, as benchmarked in the MVTec AD paper. It shows how much any decomposition buys over $|x - \hat{x}|$. It is run with that paper's protocol on our splits.
Diffusion reconstruction with partial noising. This is the noise-then-denoise approach analyzed by Bercea et al. It is the most direct competitor because it uses the same normal-only diffusion model we use. We run it with the identical trained model, and we give its noise depth the same tuning budget, in validation evaluations, that $\lambda$ receives. In one condition it is tuned on normal-only validation data, as in our method. In a second condition it is tuned with oracle access to anomalous validation images, which gives it an upper bound. No reported result was found on Lacuna for this method on MVTec AD, VisA or BraTS, so we produce its numbers ourselves. Beating it means higher pixel-level AUPRO and pixel AP at matched tuning, together with a visibly flatter sensitivity curve.
Synthetic-anomaly methods. The strongest existing methods are TransFusion and DRÆM. On MVTec AD, TransFusion reports image AUROC 99.2% and localization AUPRO 91.6%, and on VisA it reports image AUROC 98.5% (TransFusion). DRÆM reports image AUROC 98.0 and pixel AP 68.4 on MVTec AD (DRÆM). We use the released configurations, retrain them where needed on our exact splits, and report reproduced alongside published numbers.
We do not expect to beat TransFusion's averages on MVTec AD. Our success criteria are narrower. On MVTec AD we aim to stay within a few points of TransFusion's averages without ever training on synthetic defects. We aim to beat both methods on the subset of MVTec AD and VisA categories whose defects are missing or misplaced components. We also aim to beat both on BraTS 2018 lesion localization. For BraTS, the file reports only Xia et al.'s pseudo-healthy quality: identity MS-SSIM 0.88 and healthiness 0.99, in their paired setting. No localization number was found on Lacuna, so we run all methods ourselves.
Ablations. Two ablations isolate the key ingredients. The first removes the explicit anomaly layer: $a$ is fixed to zero and the anomaly map is taken as $|x - n|$ after the same guided diffusion. This tests whether the separation itself matters, as opposed to merely guiding diffusion with the input. The second toggles the exclusion penalty on and off, with results reported separately for additive anomalies (scratches, stains, contamination) and subtractive ones (missing parts).
Why not just…?
Why not just tune the noise depth carefully? Bercea et al. argue that no single depth suits both small and large anomalies, because anomaly size and contrast vary within one test set. A per-dataset tuned depth may nonetheless come close on average. That is a real threat, and the matched-budget oracle-tuned condition is designed to settle it. If the tuned depth matches us on average and on sensitivity, the method has no advantage, and we will say so.
Why not just use TransFusion or DRÆM, which already separate normal and anomalous appearance? On MVTec AD they probably will win, since their synthetic defects resemble its scratches and stains. Their separation, however, is learned entirely from a hand-designed defect generator. The per-type split and the BraTS experiment test whether a test-time separation trained without that generator generalizes better where the generator's assumptions break. If it does not, the proposal reduces to "slower DRÆM without augmentation," and the result is negative.
Why not just add the exclusion loss to DRÆM's restorer? This is cheap, and it is exactly the incremental version of the idea, so we run it as an extra baseline. It leaves DRÆM's dependence on synthetic defects intact, and it inherits the subtractive-anomaly problem we predict for exclusion. If DRÆM with exclusion matches our full method, the contribution is the loss, not the decomposition.
Why not just apply robust subspace recovery in a latent space, as RSRAE does? RSRAE assumes normal samples lie on one linear subspace and separates whole samples, not pixels. It has no mechanism for localization on multimodal textures. We include it only for image-level scores.
Proposed Experiments and Decisive Checks
We evaluate on MVTec AD and VisA with image AUROC, pixel AP and AUPRO. We evaluate on BraTS 2018 with Dice and pixel AP for lesion localization, plus MS-SSIM between $n$ and the healthy regions of the input to check that anatomy is preserved.
Three results decide the proposal. First, the sensitivity curves must show that localization holds over a wide band of $\lambda$ in settings where diffusion reconstruction holds only over a narrow band of noise depth. If the curves look alike, the core claim fails. Second, the ablation that drops the anomaly layer must lose clearly. Otherwise the gain came from guidance, not from layer separation. Third, the per-type breakdown must show the predicted sign flip of the exclusion penalty: it should help on additive defects and hurt on missing parts. Confirming or refuting that flip tells us whether reflection-style edge priors transfer to defects at all, which neither field has yet tested.
References
- Single Image Reflection Separation With Perceptual Losses
- Denoising Diffusion Models for Anomaly Localization in Medical Images
- Adversarial Pseudo Healthy Synthesis Needs Pathology Factorization
- MVTec AD - A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection
- TransFusion - A Transparency-Based Diffusion Model for Anomaly Detection
- DRÆM - A discriminatively trained reconstruction embedding for surface anomaly detection
- Benchmarking Single-Image Reflection Removal Algorithms
- Single Image Layer Separation Using Relative Smoothness
- Separate in Latent Space: Unsupervised Single Image Layer Separation
- Robust Subspace Recovery Layer for Unsupervised Anomaly Detection