Reconstruction-based anomaly detection trains a generative model only on normal images. At test time it "repairs" an input back toward normality and flags whatever had to change. This works well for scratches and dents. It fails on logical anomalies, where every part looks fine but the arrangement is wrong: a missing screw, an extra nut, two pushpins in one compartment, a wire attached to the wrong slot. Pixel-level repair cannot catch these, because each local patch is already normal. We propose to move the repair step out of pixel space and into a space that image generators adopted long ago to keep multi-object scenes coherent: an explicit layout, meaning a set of object entries, each carrying a category, a position, and a shape. We train a diffusion model (a generator that learns to turn noise into samples by repeated denoising) over these object sets from normal images. To score a test image, we partially noise its layout and denoise it back onto the normal layout distribution. The objects the model deletes, adds, or moves are the anomaly, and they come with a location, including the location where a missing part should have been.
This matters for two reasons the current leaders on MVTec LOCO AD do not address. The first is localization. The best systems either score composition with a discriminative network or match against histograms and memory banks, so they say that an arrangement is wrong more readily than where the fix belongs. The second is robustness to harmless motion. A continuous model of where objects normally sit should separate a part that has shifted a few pixels from a part that is truly misplaced. Fixed grid histograms cannot make that separation.
Core Claim
A denoising diffusion model over sets of object tokens, trained only on normal scenes, acts as a projection onto the manifold of valid multi-object layouts. The difference between the observed set and its projection, matched object to object, gives a logical anomaly score and a localization map. On logical anomalies this should beat both hand-designed count and grid statistics and discriminative composition scoring, particularly at the pixel level and under nuisance shifts. The new element is to perform reconstruction-based detection's "anomalies as noise" projection, as in Lu et al. and Bercea et al., on a structured layout of discrete entities rather than on pixels or a flat feature grid.
Where This Sits
Two bodies of work supply the method. From multi-object generation we take the lesson that coherent scenes require separating where from what. Decomposing Image Generation into Layout Prediction and Conditional Synthesis predicts a semantic layout before rendering. Image Generation From Scene Graphs computes per-object boxes and masks from a relational graph. Image Synthesis From Reconfigurable Layout and Style shows that boxes can be moved without disturbing other objects. Those papers generate layouts from a specification. We instead learn the distribution of layouts from normal data and use it to correct observed ones. From reconstruction-based detection we take diffusion-as-projection and the warning about the identical shortcut. That warning, made concrete in DRÆM, is that a high-capacity reconstructor can simply copy its input, anomaly included.
The closest prior work motivates and constrains the design. SALAD builds unsupervised component segmentation maps from SAM-HQ and DINO pseudo-labels. It then trains a composition branch discriminatively on synthetic mutations of those maps, reaching 96.1% mean image AUROC on MVTec LOCO AD. Its authors leave interaction between branches open. Our composition model is generative and object-structured rather than a classifier over 2D maps, and it outputs a corrected layout rather than only a score. CSAD gets its pseudo-labels from RAM++ and Grounded-SAM and scores logical errors with fixed-grid patch histograms, reaching 95.3% AUROC. Its grid is exactly the brittleness under shifts that continuous coordinates should remove. PSAD uses few-shot labeled part segmentation with class-histogram and composition memory banks, reporting 98.1% logical AUROC. That figure shows count statistics are already very strong, which constrains how much headroom we can claim at the image level. ULSAD and GLCF capture global logic implicitly, through attention-map reconstruction and a ViT token bottleneck respectively, with no discrete objects. LogicAL synthesizes logical anomalies by editing edge maps. Its authors note that edges lose semantic identity, and its reported 69.7 AU-sPRO is the localization number we target.
We deliberately do not build the most obvious combination: scene-graph extraction, then graph correction, then layout-conditioned image inpainting, then pixel residual. A reviewer would rightly ask what image rendering adds once the layout is already corrected. LOCO's logical ground truth marks regions such as the place a missing object belongs, and a corrected layout gives that region directly. Rendering pixels would bring back blur and GAN artifacts, which GLCF explicitly warns against, while providing no extra information about the logical anomaly. Pixel rendering therefore appears only as an ablation.
Research Program
Object tokens. Each training image is parsed into objects using the same unsupervised pseudo-labeling chain as CSAD or SALAD, so that this front end is held constant against both. Each object becomes a token: a category vector, a box center and size, a compact mask-shape code, a small appearance embedding pooled from DINO features inside the mask, and an "exists" flag. Every image is padded to a fixed number of slots. Padding with an exists flag lets the model represent missing and extra objects as ordinary values, so it never has to change the number of slots.
The model. A permutation-equivariant transformer, meaning one whose output does not depend on the order in which objects are listed, is trained as a denoising diffusion model over these token sets from normal images only. Continuous fields receive Gaussian noise. Category and existence fields receive discrete corruption that flips them toward a uniform prior. Attention between tokens is where relational rules are learned, for example that each pushpin compartment holds one pin, or that connector wires share a color. The layout generators above use explicit graph edges for this. We will add explicit predicate edges as a variant, but we do not assume them.
Scoring and localization. For a test layout $x$, we noise it to an intermediate level $t$ and denoise it $K$ times to obtain projections $\hat{x}^{(k)}$. Each projection is matched to $x$ by Hungarian assignment. The per-object residual $r_i$ combines the change in existence, the category disagreement, and the box displacement measured in units of the normal variance learned for that object. The image score is
$$s(x) = \frac{1}{K}\sum_{k=1}^{K} \max_i \; r_i\big(x, \hat{x}^{(k)}\big),$$
where $i$ ranges over matched slot pairs. The pixel map paints each $r_i$ into the corresponding mask. A slot that the projection adds is painted into the projected box, which marks where the missing part should be. The noise level $t$ is the key control against the identical shortcut. Too little noise and the model copies the anomaly. Too much and it forgets the instance. We choose $t$ on held-out normal images by the trade-off between reconstruction stability and diversity, never using anomalies.
Structural defects are handled by an unchanged local appearance branch, EfficientAD as in SALAD. We fuse with SALAD's own rule. This lets us replace exactly one component, the composition branch, and attribute any difference to it.
Baselines
The naive reference point is a non-learned layout statistic computed on our identical object tokens: per-category counts and centroid positions modeled with a per-category Gaussian, scored by Mahalanobis distance (distance measured in units of the normal spread), plus CSAD-style fixed-grid histograms on the same masks. This is the baseline most likely to embarrass us. PSAD's 98.1% logical AUROC with histogram memory banks suggests that counts carry most of the image-level signal. To beat it, we must exceed it on logical image AUROC in at least the relational categories, Splicing Connectors and Pushpins, and clearly on logical localization. If we only tie it, the conclusion is that the diffusion machinery is unnecessary.
The strongest existing method is SALAD, reported at 96.1% mean image AUROC on MVTec LOCO AD in SALAD. We will rerun its released configuration and then swap its composition branch for ours, keeping the EfficientAD and global branches, the fusion rule, the pseudo-label source, the training images, and an equal tuning budget matched in GPU hours. Beating it means a higher logical-subset image AUROC and higher logical AU-sPRO at equal or better structural performance. For localization, the reference is LogicAL's 69.7 pixel AU-sPRO from LogicAL. We also report CSAD (95.3% AUROC in CSAD) and ULSAD (84.1% in ULSAD), quoted from their papers and rerun where code allows.
Three ablations isolate the key ingredient. The first trains the same set transformer as a plain autoencoder with no noise. We expect a gap here that measures the identical shortcut. The second trains the same transformer as a discriminator on synthetic set mutations (deleting, duplicating, and displacing tokens), which reproduces SALAD's discriminative idea in our representation. It separates "object tokens help" from "generative projection helps." The third adds layout-conditioned image inpainting with pixel residuals, to test our claim that rendering adds nothing to logical localization.
Why not just…?
Why not just count objects and threshold the histogram? On categories where the rules reduce to cardinality, such as Screw Bag, this may well match us, and PSAD's 98.1% logical AUROC is evidence that it is strong. The naive Gaussian-on-counts baseline sits on identical masks precisely so that this question is answered rather than argued. Our expected advantage is confined to relational rules that counts cannot express, such as matching connector colors or one pin per compartment, and to localizing where a missing object belongs. If the per-category results show no gap on those categories, we will report that layout counts suffice.
Why not just train a discriminator on synthetic composition mutations, as SALAD does? This is the real threat, since SALAD already reaches 96.1%. A discriminator can only reject mutations resembling those it was trained on, and it outputs a score without a corrected layout. A projection needs no anomaly model and returns the repair. The second ablation runs both ideas in the same token space. If the discriminator matches us on unseen logical anomaly types and on localization, the generative framing has not earned its cost.
Why not just run pixel-space diffusion reconstruction on the whole image? Pixel reconstructors either copy locally normal parts, which is the identical shortcut DRÆM documents, or blur high-frequency edges into false positives, the failure GLCF cites in avoiding image reconstruction. A missing component leaves no local evidence for a pixel model to "fix." We include a pixel-diffusion baseline with a matched noise budget to show this directly on the LOCO logical subset.
Why not just use larger patches or global pooling with PatchCore? ULSAD reports PatchCore at 81.49% image AUROC on LOCO with roughly 7 GB of memory. Enlarging patches does not make distant parts aware of each other's count or identity, so we treat this as a reported reference rather than a primary competitor.
Proposed Experiments
The main benchmark is MVTec LOCO AD. We report image AUROC split into logical and structural subsets and per category, plus pixel AU-sPRO, again split by subset. MVTec AD and VisA serve as checks that swapping the composition branch does not hurt structural detection. SALAD reports 98.9% and 97.9% on these, and we must stay within noise of those figures.
Three decisive checks would settle the claim. The first is logical localization. The proposal stands or falls on logical AU-sPRO beating both SALAD's composition branch and LogicAL's 69.7, with the gain concentrated on missing-object anomalies, where only a projection can point at empty space.
The second is a nuisance-shift stress test. We apply small rigid translations and rotations to normal LOCO test images, within ranges that leave every logical rule intact, and measure false-positive rate at a fixed true-positive rate. CSAD's grid histograms should degrade and our continuous layout model should not. No reported result was found on Lacuna for this protocol, so all methods will be run by us under identical perturbations.
The third is the noise-level curve. Sweeping $t$ should trace the predicted shape: logical AUROC near the autoencoder ablation at small $t$, a peak at intermediate $t$, and a decline at large $t$ as projections lose the instance. If the curve is flat, the projection idea is doing no work and the gains come from the tokens alone.
Pseudo-label errors propagate in SALAD and would propagate here as well. We will therefore also report results with a deliberately degraded segmenter to measure how gracefully each method fails.
References
- Removing Anomalies as Noises for Industrial Defect Localization
- Denoising Diffusion Models for Anomaly Localization in Medical Images
- Decomposing Image Generation into Layout Prediction and Conditional Synthesis
- Image Generation From Scene Graphs
- Image synthesis from reconfigurable layout and style
- DRÆM - A discriminatively trained reconstruction embedding for surface anomaly detection
- SALAD -- Semantics-Aware Logical Anomaly Detection
- CSAD: Unsupervised Component Segmentation for Logical Anomaly Detection
- Few Shot Part Segmentation Reveals Compositional Logic for Industrial Anomaly Detection
- Revisiting Deep Feature Reconstruction for Logical and Structural Industrial Anomaly Detection
- Learning Global-Local Correspondence with Semantic Bottleneck for Logical Anomaly Detection
- LogicAL: Towards logical anomaly synthesis for unsupervised anomaly localization