Shared Nuisance, Separate Content: Reconstruction Anomaly Detection Under Intra-Class Drift

Reconstruction-based anomaly detection trains a model only on defect-free images. At test time the model repaints each input as its nearest normal version, and the difference between input and repaint marks the defect. When one model has to serve many product categories at once, two failures pull against each other. Similar-looking categories bleed into each other, so a carpet gets repainted with wood grain. The usual remedy is to confine each category's reconstruction to a small learned vocabulary of normal patterns. That remedy breaks when a single category varies continuously in lighting, pose or color, because every normal image slightly off the vocabulary lights up as a false defect.

We propose to split what the reconstruction is allowed to share across categories. The content, meaning what a normal part of this category looks like, is kept category-specific and forced through a discrete vocabulary. The nuisance, meaning smooth lighting fields, small warps and color drift, is modeled by one small network shared by all categories. That network is limited so tightly that it can bend the template but cannot paint a local defect. The claim to test is this: the right boundary between "shared" and "separate" in multi-class anomaly detection is nuisance versus content, not category versus category. Getting it right should remove the false alarms that intra-class variation causes without reopening the shortcut where the model simply copies its input.

Where this sits

Existing multi-class reconstruction models occupy the two extremes, and both motivate the proposal. A Unified Model for Multi-class Anomaly Detection (UniAD) ignores category identity. It blocks the "identity shortcut", in which a high-capacity decoder learns to copy its input and so reproduces the defect too, by masking each location's neighbors in attention. Nothing in UniAD stops one category's patterns from reconstructing another's. At the other extreme, Toward Multi-class Anomaly Detection: Exploring Class-aware Unified Model against Inter-class Interference (MINT-AD) conditions the decoder on category through a modulation network. Its authors say it "does not establish connections between relevant classes." Revitalizing Reconstruction Models for Multi-class Anomaly Detection via Class-Aware Contrastive Learning (CCL) treats every other category as a negative in its contrastive loss. Both treat anything shared between categories as interference. We keep their separation for content and depart from them by sharing the nuisance model.

The prototype line supplies our content mechanism and also constrains it. Hierarchical Vector Quantized Transformer for Multi-class Unsupervised Anomaly Detection (HVQ-Trans) forces features through category-routed discrete codebooks, a fixed set of learned normal feature vectors. It reports that removing this quantization step costs roughly 26% AUROC, which is strong evidence that a discrete bottleneck is what defeats the shortcut. Vague Prototype-Oriented Diffusion Model for Multi-Class Anomaly Detection (VPDM) low-pass filters features and matches them to 50 prototypes before a diffusion model restores detail. Its authors flag the fixed prototype budget and a sensitive filter setting as limits. Both designs ask the codebook to absorb all variation, including continuous drift. The remote-sensing and pathology literature describes exactly this kind of drift: seasonal and illumination change within one land-cover class (Sumbul et al.) and patient-to-patient tissue variation (Tommasi et al.). Our move is to take continuous variation out of the codebook entirely. That differs from VPDM's frequency heuristic, which discards high-frequency detail rather than modeling the variation.

ReContrast adapts reconstruction to medical and industrial domains, one model per category. It shows that unconstrained end-to-end training collapses, so that anomalies get reconstructed too. This negative result constrains us: the nuisance path must stay low-capacity by construction, not by early stopping.

Core mechanism

We reconstruct in the feature space of a frozen pretrained backbone, following the reverse-distillation practice the file describes. Let $F(x)$ be the backbone feature map of image $x$. Let $c$ be its category, taken either from labels or from clustering pseudo-labels, which CCL found roughly as good. Let $Q_c$ be a category-routed discrete codebook lookup in the style of HVQ-Trans that turns $F(x)$ into a normal content template. A single shared nuisance network $\phi$ reads the input and outputs three things. The first is a warp $W_x$ defined by a coarse grid of control points. The second and third are a gain field $A_x$ and a bias field $B_x$ over feature channels, each restricted to at most $K$ low-frequency spatial modes. The reconstruction and per-location anomaly score are

$$\hat F(x) = A_x \odot W_x\!\left[Q_c(F(x))\right] + B_x, \qquad s_u(x) = 1 - \cos\!\left(F(x)_u, \hat F(x)_u\right),$$

where $u$ indexes spatial locations and $\odot$ is elementwise product.

The key property is that $W_x$, $A_x$ and $B_x$ are band-limited, so they have too few degrees of freedom to reproduce a localized scratch or missing component. Any local structure in the reconstruction must come from the category's codebook. The number of modes $K$ and the control-grid resolution are the single dial that trades false positives from unmodeled drift against leakage through the nuisance path. We expect performance to peak at an intermediate bandwidth, and we expect that peak to move upward as nuisance variation in the data grows. Sharing $\phi$ across categories is the part of the design that crosses between the two directions. Lighting and pose drift obey roughly the same physics whether the object is a capsule or a transistor. A category with few training images under some lighting condition can therefore borrow the nuisance model learned on other categories, while its content stays isolated.

Training minimizes the cosine residual on normal images only, plus the standard codebook commitment loss. We add one consistency term: the same image under two synthetic photometric and geometric perturbations must map to the same codebook entries. This pushes variation out of content and into $\phi$. The term uses only mild global transforms, never local ones, which heeds ReContrast's warning that augmentations resembling defects confuse the objective.

Research program

The work has three parts. First we build the model on top of the public HVQ-Trans design. Keeping the codebook identical isolates the effect of the nuisance path. Second, we build a controlled stress test, called MVTec-Drift here. MVTec AD's normal training and test images, and its anomalous test images, receive the same distribution of smooth illumination gradients, color casts and small rotations or warps at a graded magnitude level. This recreates high intra-class variation on a benchmark whose clean version is near saturation, and it lets us plot detection quality against drift magnitude. Third, we evaluate on benchmarks with natural intra-class variation. These are VisA (unaligned and multi-instance objects), MPDD (variable lighting and backgrounds), Real-IAD (30 categories, multiple viewpoints) and BTAD. For medical transfer we use APTOS and ISIC2018.

Following MVTec AD, which showed image AUROC of 0.86 alongside per-region overlap of 0.05 on the cable category, and following SPot-the-Difference, which argues that precision-recall suits imbalanced inspection better, we report four metrics. These are image AUROC, pixel AUROC, per-region overlap (PRO) and pixel AU-PR. We also report the false-positive rate on drifted normal images at a fixed 95% defect recall, the quantity our claim is actually about.

Baselines

The naive reference point is HVQ-Trans trained and tested unchanged on each benchmark. It is the same codebook and decoder we build on, without the nuisance path. Beating it is the minimum requirement. It also establishes how much false-positive mass drift creates when the codebook must absorb everything. HVQ-Trans reports detection AUROC of 98.0% on unified MVTec AD and 93.2% on unified VisA.

The strongest existing reconstruction method in the file is VPDM, which reports image AUROC of 98.4% and localization AUROC of 97.8% on unified MVTec AD, and 94.2% image AUROC on unified VisA. It is the most dangerous competitor because its low-pass filtering is itself a crude way of discarding nuisance-like variation, which is a cheaper answer to the same problem. We will run VPDM from its public configuration with its reported prototype count, and give it the same tuning budget we give ourselves: a sweep of its filter cutoff matching our sweep of $K$. Every method gets the same backbone family where the architecture allows, the same training images, and the same drifted data on MVTec-Drift.

UniAD (96.5% image and 96.8% pixel AUROC on unified MVTec AD) and PatchCore (96.4% AUROC on unified MVTec AD, as reported in HVQ-Trans) are included as the category-agnostic and non-generative references. MINT-AD is included as the fully separated class-conditioned reference. For MINT-AD, and for MPDD, BTAD and Real-IAD generally, the file gives no reported numbers: no reported result found on Lacuna for those settings. We will therefore rerun all methods ourselves. On APTOS and ISIC2018 the reference is ReContrast one-for-one, which reports 97.51% and 90.15% image AUROC. A unified model matching these would be notable, because it serves all categories with one set of weights.

Beating the field means two things. On clean MVTec AD we must stay within noise of VPDM. On MVTec-Drift at medium and high drift, and on VisA, MPDD and Real-IAD, we must beat both VPDM and HVQ-Trans in PRO and pixel AU-PR, with a lower drifted-normal false-positive rate.

Three ablations isolate the key ingredient. In the first, the nuisance network is replaced by a separate one per category, which tests whether sharing matters. In the second, the nuisance network is unconstrained, with full-resolution fields, which should reproduce the identity shortcut and show that the band limit is load-bearing. In the third, the nuisance path is removed and the codebook is enlarged to use the same added parameter count, which tests whether we have simply added capacity.

Why not just…?

Why not just train HVQ-Trans or VPDM with heavy photometric and geometric augmentation? This is the most serious threat. Augmentation also teaches invariance to drift, at zero architectural cost. Our reason to doubt it suffices is that augmentation makes the codebook cover a wider manifold, which spends its fixed prototypes on nuisance. VPDM already notes that its prototype budget is a limit, and a widened normal manifold is also what lets anomalies pass through. We will not argue this point, we will measure it. The augmented-baseline comparison on MVTec-Drift uses exactly the transforms that generate the drift, and is therefore generous to the baseline. If augmentation closes the gap at matched codebook size, the explicit nuisance path is unnecessary and we will report that.

Why not just train one model per category? Separate models remove cross-category bleed, but they cannot share a nuisance model. UniAD and CCL document the checkpoint cost of dozens of categories. More to our point, a per-category model still has to absorb its own category's continuous drift in its codebook. Our subsampling experiment, which reduces one category's training set while keeping others full, tests whether the shared nuisance path helps data-poor categories more than an isolated model can.

Why not just use a larger or deeper codebook? More prototypes can tile a continuous drift manifold, but tiling needs more entries as drift grows, and a larger codebook also gets closer to covering defects. The matched-parameter ablation answers this directly. If a bigger codebook matches us, then the nuisance-content split adds nothing.

Why not just use nearest-neighbor retrieval like PatchCore? Retrieval avoids decoders altogether. However, drifted normal patches sit far from the stored ones in exactly the same way as defects. For lookalike categories, a defect in one category can also find a spurious normal match in another category's memory, and retrieval produces no repaired counterfactual image. PatchCore runs on MVTec-Drift alongside everything else. Its drifted-normal false-positive rate shows whether this concern holds.

Decisive checks

Four results would decide the idea. First, the gap between our model and HVQ-Trans or VPDM must widen as MVTec-Drift magnitude increases and be near zero at no drift. A flat gap falsifies the claim. Second, the bandwidth sweep must show an interior optimum that moves upward with drift magnitude. A monotone curve would mean the band limit is not doing the work described. Third, a leakage probe inserts synthetic local defects of known size and measures the fraction of each defect's residual that the nuisance path absorbs as $K$ grows. This number must stay small at the chosen $K$. Fourth, a cross-category camouflage test pastes texture from one category into another as the defect, for example wood into carpet. Category-specific content should flag these, whereas we expect UniAD to miss some. This test confirms that sharing nuisance has not reopened the inter-class bleed the separated designs were built to stop.