Paired Texture Counterfactuals for Auditing Industrial Anomaly Detectors

Industrial anomaly detectors are trained only on photos of defect-free parts and asked to flag and outline defects at test time. On the standard MVTec AD benchmark they look nearly solved; SimpleNet, for example, reports 99.6% image-level AUROC. Yet when Texture-AD tested the same families of methods on new fabric weaves, wafer patterns, lighting and camera angles, the average fell to 55.23%. The contamination literature has a name for this pattern: a benchmark score that measures familiarity with the test distribution rather than the skill it claims to measure. In anomaly detection the familiarity is with one specific surface texture, because normal test images share the exact material seen in training. A detector can score perfectly by learning "anything that looks unlike this carpet is a defect." That rule fails the moment a benign texture change arrives. It also accounts for two failures the texture literature describes separately: false alarms on harmless surface patterns, and defect maps that break into many disconnected fragments along texture ridges.

We propose a controlled test that separates two abilities that current benchmarks mix together. The first is detecting any departure from the training texture. The second is detecting a break in the surface's structure. The test pairs two kinds of edits to real, pixel-exact images. A benign edit changes the texture but not its integrity; it should not fire. A structural edit keeps the local texture statistics but breaks the pattern's regularity; it should fire. This is the anomaly-detection analogue of the counterfactual-perturbation test that Contamination Detection for VLMs used to expose memorization in vision-language models. Their version relied on diffusion editing, which they note struggles with fine surface micro-texture. We instead build every counterfactual by rearranging and blending the dataset's own pixels, so no generative artifacts appear.

Core Claim

Detectors whose training teaches them that "foreign texture means anomaly" will show high benign-edit false-alarm rates even when their MVTec scores are near perfect. That group includes DRAEM and Adaptive Deviation Learning, both of which paste textures from the Describable Textures Dataset (DTD) as fake defects. Our paired score, measured on MVTec AD alone, should predict how a method ranks on Texture-AD's real shifts better than its MVTec AUROC does. A second, conditional claim follows if the first holds. The false alarms should shrink if a detector is trained with two changes: benign edits labeled as normal, and structural edits (rather than DTD pastes) as the synthetic anomalies. The same change should also reduce fragmentation of real defect maps.

Research Program

The instrument has two edit families, each with a dose knob, and both are built only from a category's own training images.

Benign edits come in four kinds. The first resamples the texture layout: patch-based texture quilting from other training images of the same class gives a new arrangement with identical micro-structure. The second applies smooth illumination gradients. The third applies small global rotations and rescalings. The fourth blends in a sibling material; for Texture-AD we use a neighbouring subclass. That blend is controlled by a coefficient $\eta \in [0,1]$, borrowed from the controlled texture replacement in TextureSAM. TextureSAM used this knob to teach a segmenter to separate shape from texture. We use it to sweep how far a surface can drift before a detector calls it defective.

Structural edits keep local statistics but break global regularity. For periodic textures (grid, carpet, woven cloth, wafer) the edit is a lattice dislocation: a region is shifted by a fraction of the pattern period, or a row is duplicated, so every local patch still looks normal while the pattern is broken. For stochastic textures (leather, wood) the edit is a local change in grain orientation or scale inside an irregular mask. These edits are dosed by shift magnitude and region size. They come with exact pixel masks for free.

Each detector is thresholded on clean held-out normals at a fixed false-positive rate. We then record four quantities. The first is the benign-edit false-alarm rate as a function of $\eta$ and edit type. The second is AUROC between benign-edited normals and structurally edited anomalies, which we call the paired AUROC. The third is region overlap (PRO) on structural edits and real defects. The fourth is a fragmentation count: the number of predicted connected components per ground-truth defect region, plus false-positive area lying on texture ridges.

The fragmentation count makes texture-induced over-segmentation measurable in defect localization. The MVTec AD paper already showed how far image-level and region-level scores can diverge: cable reached 0.86 AUROC but only 0.05 region overlap (MVTec AD).

We use MVTec AD texture categories, DAGM, and Texture-AD. All methods run in the shared ADer toolbox where available, so backbones, resolution and training budget are identical.

The intervention stage runs only if the audit shows DTD-trained detectors failing on benign edits. It keeps the DRAEM and SimpleNet architectures unchanged and changes only the training signal. Benign edits enter as extra normals, and structural edits replace DTD pastes (for DRAEM) or noise-injected features (for SimpleNet) as the synthetic anomalies. To avoid circularity, the structural edits used for training are drawn from different families than those used in the audit: dislocations for training and orientation breaks for testing, then swapped.

Baselines

The naive reference point is each method's MVTec AD AUROC used as the predictor of Texture-AD performance. The audit has to beat this reference in rank correlation with Texture-AD per-subclass scores, or it adds nothing. The external numbers we anchor to come from Texture-AD: SimpleNet reaches 64.41% I-AUROC on Cloth, DRAEM 56.12% on Wafer, EfficientAD 63.30% on Metal Plate, and the average is 55.23%. MVTec AD reports 99.6% I-AUROC for SimpleNet.

The strongest method-level competitors are three. SimpleNet adds Gaussian noise to adapted pretrained features as its fake anomalies, and it is the strongest reported MVTec number in the file. SingleNet is a one-layer teacher-student model with Fourier convolutions. Its whole-image receptive field should in principle tolerate global texture shifts, so it is the most likely method to already pass the benign test. HarmoniAD splits CLIP features into high- and low-frequency paths and reports a pixel-level precision-recall of 58.3 on MVTec AD. It is the method that explicitly targets the structure-versus-semantics trade-off.

For the intervention, the strongest baseline is plain photometric and geometric augmentation (colour jitter, rotation, scale) applied to normals at the same compute budget. The second is few-shot adaptation to the target subclass with InCTRL, given the same handful of target normals. The third is the shift-aware training of ADShift. Every method gets the same hyperparameter search budget on clean validation normals only.

The key ablation isolates the two ingredients by crossing {DTD pastes, structural edits} with {with, without benign normals}. Beating the baselines means lower benign false-alarm rates at a matched clean false-positive rate, with no loss of MVTec pixel AUROC beyond one point, and higher Texture-AD I-AUROC than the augmentation baseline. There is no reported result on Lacuna for any detector's benign-edit false-alarm rate or fragmentation count, so those numbers will be produced here for all methods.

Why not just…?

Why not just evaluate on Texture-AD, which already shows the collapse? Texture-AD is our validation target, not a substitute. Its shifts combine weave, lighting and viewpoint changes at once. It cannot say whether a method fails because it fires on benign texture change or because it misses defects on the new material, and those failures call for opposite fixes. The instrument's value depends on its MVTec-only scores predicting Texture-AD rankings. If they do not, the audit is redundant and we will report that.

Why not just add standard augmentation to the normal training images? This is the most serious threat to the intervention. Augmentation may remove most benign false alarms by itself. It is therefore a head-to-head baseline in the ablation. The claim that benign edits must be paired with structural anomalies survives only if augmentation alone lowers false alarms while also blurring structural sensitivity, which the paired AUROC is designed to detect. If augmentation matches on both measures, the contribution reduces to the audit.

Why not just smooth the anomaly map to remove fragments and speckle? Post-hoc filtering is cheap. We will apply Gaussian and morphological cleaning to every method as a variant. SingleNet's own low-pass filter still fails on randomized DAGM textures, and HarmoniAD's ablation finds fixed frequency cut-offs inferior to adaptive ones. So we expect smoothing to trade small structural edits for fewer fragments. The dose curves over region size will show whether it does.

Why not just use a few normal images from the new subclass? Few-shot adaptation answers a different deployment question: it assumes target normals exist. It is included as a baseline. The audit asks whether a detector degrades gracefully before any such data is collected.

Decisive Checks

The first check is predictive validity. Across roughly 39 Texture-AD subclasses times six or more methods, is the rank correlation between MVTec-only paired AUROC and Texture-AD I-AUROC clearly higher than the correlation for MVTec AUROC? We will report bootstrap confidence intervals, since the number of methods is small.

The second check is mechanism. Do DRAEM and ADL show steeper benign false-alarm curves in $\eta$ than SimpleNet, SingleNet and HarmoniAD? Swapping DTD pastes for structural edits in DRAEM should flatten its curve, and the remaining metrics will show at what cost.

The third check is transfer to real defects. The intervention should lower fragmentation counts and raise region overlap on real MVTec and Texture-AD defects, not only on our synthetic edits.

A further contamination test follows. The benign normals from the audit are used to re-examine ADL's sample reweighting, which the authors report drops 9–13% on cables, transistors and screws. The question is whether that reweighting mistakes complex normals for training-set defects.