How Much of Zero-Shot Anomaly Localization Is Just Knowing Where Defects Usually Are?

Zero-shot anomaly detection finds defects in product images without training on that product. Systems like WinCLIP compare image patches against text such as "a photo of a scratched screw" and report pixel-level scores on the MVTec-AD and VisA industrial benchmarks. Those scores are measured with pixel AUROC and region-overlap metrics that ask whether defect pixels outscore normal pixels anywhere in the image. On these benchmarks the product sits in the middle of the frame on a plain background, and defects only ever occur on the product. A predictor that outputs nothing but "the object, brighter toward the middle" may therefore score well without detecting anything. Eye-tracking research met exactly this problem decades ago. There, a static center blob predicts human fixations well, and the field responded with shuffled metrics and information-gain-over-prior scoring. The same research area also supplies a mechanism, inhibition of return: an attention system suppresses places it has already inspected so it moves on to new ones.

This note proposes two things. First, import prior-controlled evaluation from gaze modeling into zero-shot anomaly localization, and measure how much of each published method's score survives. Second, test whether one cheap mechanism recovers what the prior cannot supply. That mechanism is a fixed-budget loop that takes high-resolution "glimpses" at candidate locations, then suppresses each inspected spot with a slowly fading mark so the loop can come back later. The loop targets the failure that weakly supervised localization calls part collapse, where a map lights up only the single most distinctive region. Here the analogues are anomaly maps that fire on one conspicuous defect or on the whole object while missing small, peripheral, or secondary defects.

Core Claim

The central claim is that standard zero-shot anomaly localization numbers mix three things: detecting defects, segmenting the foreground object, and following the center and position prior of the dataset. Once the last two are held fixed, method rankings change. The size of the change depends on the method. Pipelines that explicitly mask to the object, such as FiLo's Grounding DINO step, should lose the most. Batch-comparison methods that ignore position, such as MuSc, should lose the least.

The second, conditional claim is about compute. At an equal budget, glimpses placed by a leaky inhibition-of-return schedule recover more prior-controlled localization than uniform high-resolution tiling does. We treat the first claim as the primary contribution, because it is cheap, falsifiable, and useful whatever the second claim's outcome.

Relation to Prior Work

The motivation comes from two literatures that do not cite each other.

In saliency evaluation, Analysis of Scores, Datasets, and Models in Visual Saliency Prediction showed that uniform-negative AUC rewards a center blob. It standardized shuffled AUC, which draws negatives from fixations on other images so that the center blob scores 0.50. Spatially Binned ROC evaluates ROC separately in concentric rings so that weakness in the image periphery shows up. Using Deep Features to Predict Where People Look scores models by the information they add beyond an explicit center-bias baseline. None of these has been applied to defect localization; we reuse their logic directly.

On the architecture side, When Random Saliency Looks Trained shows that zero-padded CNNs produce center-heavy maps even with random weights, while vision transformers stay near neutral. It also introduces a corner-shift test, in which the correlation between trained and random maps falls from about 0.40 to 0.15. ImageNet-Hard finds that CLIP, unlike standard classifiers, does not prefer center crops.

These two results constrain our hypothesis. If the confound exists in zero-shot anomaly detection, it should come mainly from dataset structure and object-masking steps rather than from the CLIP vision transformer. We therefore predict that text-scoring methods will be relatively robust to translating the object, and object-grounded pipelines will not.

The methods under audit are WinCLIP, FiLo, CLIP-SAM with cascaded prompts and MuSc:

  • WinCLIP scores dense multi-scale windows against prompt ensembles.
  • FiLo adds GPT-4 defect descriptions and masks the background with Grounding DINO.
  • CLIP-SAM turns CLIP peaks into prompts for the Segment Anything Model (SAM), with fixed negative points placed on nearby normal regions.
  • MuSc scores each patch by its distance to patches in other unlabeled test images.

All four report uniform-negative pixel AUROC, per-region overlap (PRO), or average precision. None reports a prior-controlled figure. The research file surfaced no prior-controlled evaluation of zero-shot anomaly localization, and no reported numbers for prior-only predictors on MVTec-AD or VisA.

The mechanism comes from the scanpath literature:

We differ from all of these in two ways. The quantity driving the next glimpse is a semantic anomaly score, not low-level contrast. And the inhibition mark decays at a rate set by how uncertain the score at that spot was.

The weakly supervised localization work plays a diagnostic role rather than supplying methods. MELM, SAOL and EM-Adapt fight part collapse with training-time losses or output heads. Zero-shot detection has no training step, so those fixes do not transfer. What transfers is the diagnosis: a map optimized to rank globally will concentrate on the most conspicuous evidence. Measuring coverage per defect, rather than per pixel, exposes this.

Research Program

Measuring the prior. We build three prior-only predictors for each benchmark category:

  • a fitted 2D Gaussian centered on the frame;
  • a foreground mask of the object from Grounding DINO plus SAM, with no anomaly reasoning;
  • a leave-one-image-out average of ground-truth defect masks, which is the dataset's "where defects usually are" map.

We score these, and every audited method, with three families of metrics. The first is the standard metrics as published. The second is shuffled pixel AUROC, where negatives are normal pixels sampled at locations drawn from other images' defect masks in the same category; this is the direct analogue of shuffled AUC in gaze research. The third is foreground-restricted AUROC and PRO, where negatives come only from normal pixels on the object.

We also compute information gain over the defect-location prior. Write $s(x)$ for a method's anomaly map normalized to sum to one over the image, $q(x)$ for the leave-one-out defect-location prior normalized the same way, and $D$ for the set of defect pixels. Then

$$\mathrm{IG} = \frac{1}{|D|}\sum_{x \in D} \log_2 \frac{s(x)}{q(x)},$$

in bits per defect pixel. IG is zero for a method that only knows the prior and positive only if the method puts more mass on true defects than the prior does. The form follows the information-gain metric in Using Deep Features to Predict Where People Look; it is defined as a gain over an explicit baseline so that the prior cannot earn credit.

Displaced benchmark. Finally, we construct a displaced version of MVTec-AD and VisA. Each object is cropped with its defect mask and pasted, scaled to 50%, at random off-center positions on a larger canvas. The canvas is filled by tiling background texture from the same category, following the corner-shift design of When Random Saliency Looks Trained. This separates position priors from object priors. We report scores separately by distance from the center, in the style of spatially binned ROC.

Budgeted re-inspection with leaky inhibition. We start from a coarse zero-shot map $S_0$, taken from WinCLIP-style text scoring on the downsampled image. At step $t$ the loop picks the peak of the priority map $P_t(x) = S_0(x)\,(1 - I_t(x))$. It then takes a glimpse, a native-resolution crop around that peak (MVTec images are far larger than CLIP's input), and rescores the crop. Rescoring uses both text prompts and MuSc-style mutual scoring against a small bank of same-scale crops from other unlabeled test images. The inhibition map then updates as

$$I_{t+1}(x) = \lambda_t\, I_t(x) + G_{\sigma}(x - x_t),$$

where $x_t$ is the glimpse center and $G_\sigma$ is a Gaussian footprint of width $\sigma$ matched to the glimpse size. The decay factor is $\lambda_t = \lambda_0 + (1-\lambda_0)\,m_t$, with $m_t \in [0,1]$ the margin between the glimpse's "normal" and "anomalous" prompt similarities. Confidently resolved regions therefore stay inhibited, and ambiguous ones fade back into play. This is the semantic, uncertainty-driven counterpart of leaky tagging.

The final map combines the rescored glimpses with $S_0$ elsewhere. The budget is $K$ glimpses, and we choose $K$ so that total vision-encoder compute equals a stated fraction of dense native-resolution tiling.

Baselines

The naive reference points are the three prior-only predictors above. Beating them on standard metrics is the minimum requirement. The interesting quantity is how far each method stays above them once the metrics are prior-controlled. The research file surfaced no result for these predictors on MVTec-AD or VisA; producing those numbers is itself part of the contribution.

The strongest published competitors, run from released code on identical splits and resolutions, are as follows:

  • WinCLIP. It reports zero-shot segmentation AUROC of 85.1% on MVTec-AD and 79.6% on VisA (WinCLIP).
  • FiLo. It reports pixel AUROC of 92.3% on MVTec-AD and 95.9% on VisA (FiLo). We expect it to be the most prior-dependent, because it explicitly masks to the grounded object.
  • CLIP-SAM. It reports F1-max of 48.8% and AP of 46.4% on MVTec-AD (CLIP-SAM).
  • MuSc. It reports PRO of 93.8% on MVTec-AD (MuSc). Its batch nearest-neighbor scoring does not use position, so it is the strongest competitor on the question we actually ask: localization that survives prior control. It is the baseline the re-inspection loop must beat.

We first reproduce each method's published number. Only then do we apply the new metrics, so that any drop is attributable to the metric and not to our setup. We tune no hyperparameters of any method; the zero-shot setting leaves nothing to tune.

For the re-inspection loop, the fair comparison is at matched compute, counting vision-encoder forward passes and wall-clock time on the same GPU. The main competitor there is dense native-resolution tiling scored by the same text-plus-mutual scorer. Beating it means higher shuffled AUROC, foreground-restricted PRO, and per-defect recall at equal compute, with confidence intervals from bootstrapping over images that exclude zero.

Four ablations isolate the key ingredient:

  • glimpses placed uniformly at random, and glimpses placed center-first, at the same budget;
  • inhibition of return with no decay ($\lambda_t = 1$);
  • fixed decay ($m_t$ held constant);
  • inhibition without re-inspection, i.e., non-maximum suppression on $S_0$ with no new glimpses.

The last ablation tests whether suppression alone does anything.

Why not just…?

Why not just tile the image densely at native resolution? This is the real threat. If uniform tiling at the same compute matches the leaky-inhibition loop, the loop adds nothing, and we will report that. The paper's contribution then reduces to the prior-controlled evaluation plus the finding that resolution, not attention, is what matters. The compute-matched tiling comparison is the decisive experiment. We run it at several budgets, because inhibition of return can only help when the budget is below full coverage.

Why not just apply non-maximum suppression to the existing map? Suppression on a fixed coarse map can separate multiple peaks, but it cannot add evidence the coarse map never had, such as a hairline scratch below CLIP's patch resolution. The "inhibition without re-inspection" ablation measures exactly this. If it matches the full loop, the benefit is bookkeeping rather than new looks.

Why not just trust PRO, which already normalizes per region? PRO normalizes the defect side, but its false-positive rate is still computed over all normal pixels, background included. A method that simply segments the object therefore gets easy negatives. The foreground-restricted PRO and the prior-only foreground predictor will show directly how much of the PRO gains reported by MuSc and others survive.

Why not assume vision-transformer backbones remove the center-bias problem? They largely remove the architectural part, as When Random Saliency Looks Trained and ImageNet-Hard show. We agree with this and build it in as a prediction. The confound we target lives in the benchmark's spatial statistics and in object-masking pipeline steps, not in the backbone. Rectify ViT Shortcut Learning found that trained vision transformers still use background shortcuts (50% of cases on INbreast). The displaced benchmark separates backbone behavior from pipeline behavior.

Proposed Experiments and Decisive Checks

The first check is whether the confound is real. If the defect-location prior reaches pixel AUROC near the published numbers of weaker methods on MVTec-AD object categories, the standard metric is confounded. If the prior scores near chance, it is not, and the audit result is that current numbers are trustworthy.

The second check is whether rankings change. We compare the ordering of WinCLIP, FiLo, CLIP-SAM and MuSc under standard AUROC against their ordering under shuffled AUROC, information gain, and displaced-benchmark AUROC. We report the rank correlation between orderings, separately for texture and object categories; texture categories serve as a within-study control, since defects there can occur anywhere. The hypothesis predicts that FiLo drops most and MuSc least on displaced images, especially in the outer rings.

The third check tests the collapse analogue. On images with multiple annotated defects, we measure recall of the second and third defect regions. We compare coarse maps, maps after suppression only, and maps after leaky re-inspection.

The fourth check is the compute-matched comparison of leaky re-inspection against dense tiling, random glimpses, and center-first glimpses. We plot prior-controlled PRO against encoder passes on MVTec-AD, VisA, and the displaced variants. The mechanism claim stands only if its curve lies above tiling at sub-full budgets and the uncertainty-driven decay beats both the fixed-decay and no-decay versions. Otherwise we drop the claim and keep the measurement.