Most visual anomaly detectors learn what "normal" images look like and flag whatever deviates from it. That is a statistical question. What users actually care about is a semantic one: is this harmful, defective, or suspicious? The two often disagree. A title card in a surveillance clip or a cosmetic scuff on a part is rare but harmless. A structural crack that matches the surface texture, or an intruder walking at a normal pace, looks statistically ordinary but matters a great deal. We propose to compute, for every image patch, two separate maps. The first is a statistical map of how unusual the patch is relative to normal data. The second is a semantic map of how much a frozen multimodal language model attends to that patch when it is asked about anomalies. A small calibrated rule then combines the two, so that rare-but-irrelevant regions can be switched off and ordinary-looking-but-relevant regions can be switched on.
The opportunity is that the second map already exists inside large vision-language models and is cheap to read. HeadHunt-VAD showed that a handful of attention heads in a frozen multimodal model carry reliable anomaly information. It read them without generating text, which avoids the information loss and the bias toward describing normal background that text generation introduces. HeadHunt-VAD, however, pools those heads into a single frame-level feature and throws away where the heads were looking. Our claim is that the spatial attention pattern of such heads, selected by how well they localize rather than how well they classify, is a usable semantic relevance map. Crossed with a conventional statistical map, it closes much of the gap between statistical rarity and semantic importance at negligible extra cost.
Core Claim
The statistical and semantic signals should be kept as two per-patch quantities and combined with a rule that has an interaction term. They should not be merged by masking one with the other. Masking can only suppress: it removes benign rarities but cannot recover anomalies that look statistically normal. A model that sees both axes can do both. Concretely, let $z_{\text{stat}}(p)$ be the standardized patch distance to a memory bank of normal features, computed with PatchCore, a nearest-neighbour detector over pretrained features. Let $z_{\text{sem}}(p)$ be the standardized attention mass that the selected heads place on patch $p$ from the final prompt token. We score each patch as
$$ s(p) = \sigma\big(a\, z_{\text{stat}}(p) + b\, z_{\text{sem}}(p) + c\, z_{\text{stat}}(p)\, z_{\text{sem}}(p) + d\big), $$
where $\sigma$ is the logistic function and $a, b, c, d$ are four scalars fit on a small labeled calibration set. The interaction weight $c$ is the interesting part. If the semantic map carries information beyond saliency, $b$ should be clearly positive: relevance alone raises the score. If the two axes are complementary, $c$ should be nonzero. The fitted coefficients are themselves a readable measurement of how large the gap between rarity and relevance is on each benchmark.
Relation to Prior Work
Several papers motivate this proposal by documenting the gap. Transfer-Based Semantic Anomaly Detection showed that self-supervised proxy tasks latch onto low-level geometry. Transferring pretrained semantics raised CIFAR-10 semantic anomaly detection from 41.6 to 74.6 mAP. Generative Cooperative Learning and the review in Image Anomalies document reconstruction errors firing on overlays and non-semantic noise. Learning Feature Inversion under COCO-AD warns that AUROC saturates while threshold-dependent metrics expose real failures, and we adopt its evaluation stance.
The closest existing systems each do one half of what we propose. Adversarial Discriminative Attention multiplies reconstruction error by a discriminator's class-activation map. It admits that this suppresses out-of-context anomalies in the background, which is exactly the failure an additive-plus-interaction rule is meant to avoid. FDPN for egocentric 360° video masks frames with a generic motion-and-appearance saliency network before detection. It has no notion of malicious versus benign motion, and it runs at 1.7 frames per second. CutSwap uses bottom-up LayerCAM saliency to decide where to synthesize training anomalies, and its image AUC drops from 0.955 on MVTec AD to 0.881 on VisA. That drop constrains us: purely visual saliency weakens in multi-object scenes, which is why our relevance map is language-conditioned.
VELM runs a statistical detector, draws its detection as a visual prompt, and asks GPT-4o to classify the defect. This is a one-way pipeline that needs closed-set defect definitions and an external API. We keep the statistical detector but replace the language-model call with a single forward pass whose internal attention we read. HeadHunt-VAD supplies the method we reuse: statistical head screening and robustness to prompt paraphrase. We change the object being read, from pooled head features to attention distributions over image patches, and the selection criterion, from discriminability to localization. StackCLIP and ROADS inject semantics through prompts in CLIP-style or distillation models. ROADS needs class identity at test time, and neither separates rarity from relevance as distinct quantities.
One known finding cuts against us. VELM reports that multimodal models struggle to ground tiny defects, which is why it relies on an external detector's contours. We take this seriously by design. The semantic map is only asked to supply coarse relevance at the model's patch resolution, and fine localization stays with the statistical map.
Research Program
The work has three parts. The first builds the head-selection procedure. For every attention head in a frozen open-weight multimodal model, we extract the attention from the final prompt token to the visual tokens. We do this under several paraphrases of an anomaly-oriented prompt, such as "is anything wrong in this image." We then score each head on a calibration set containing a few pixel-annotated anomalies per category. The score combines two things: per-region overlap (AUPRO) between the head's map and the ground-truth mask, and rank stability of the map across paraphrases, in the spirit of HeadHunt-VAD's robust saliency score. We keep the top few heads and average their maps into $z_{\text{sem}}$. We also report which layers the selected heads come from and whether they transfer across categories and datasets. Transfer matters because re-selecting heads per domain would weaken the practical case.
The second part builds a diagnostic evaluation that measures the gap directly, since standard benchmarks blend its two failure modes together. On each test set we use the statistical detector alone to split samples into quadrants. Normal samples above its operating threshold form the "benign rarity" set. Anomalous samples below it form the "conforming anomaly" set. The remaining samples are easy cases. On MVTec-AC we additionally use VELM's negligible-versus-critical labels. On COCO-AD, whose anomalies are whole held-out object classes inside multi-object scenes, conforming anomalies should be common.
The third part fits the combination rule and analyzes it. For video, where pixel masks are unavailable in the benchmarks we have, we pool $s(p)$ per frame and apply HeadHunt-VAD's Gaussian temporal smoothing. Video results therefore test relevance at the frame level, not localization.
Baselines
The naive reference point is the statistical detector alone: PatchCore run through the ADer toolbox with the same backbone, memory-bank size, and normal training data used inside our method. Any gain over it is attributable to the semantic axis. It also defines the quadrant split, so its benign-rarity false alarm rate and conforming-anomaly recall are the numbers we must move.
The strongest image-side competitor on the criticality question is VELM. On MVTec-AC it reports 84.0% classification accuracy with DDAD plus GPT-4o, 87.8% with oracle masks, and 89.8% on the negligible-versus-critical decision. VELM uses a proprietary model, so we run it two ways. The first is VELM's own reported numbers. The second is VELM's prompting protocol with the same open-weight multimodal model we probe, with the same reference images and defect definitions, so that backbone knowledge is held fixed. Beating it means matching or exceeding the same-backbone VELM on negligible-versus-critical accuracy without generating text and at a small fraction of its latency, while also producing pixel maps, which VELM does not.
The strongest video competitor is HeadHunt-VAD, which reports 87.03% AUC on UCF-Crime and 82.63% AP on XD-Violence. We rerun it with the same backbone and the same 1% calibration budget it describes. We count our method as better if it improves frame AUC or AP. It also counts as better if it holds those numbers while reducing false alarms in the benign-rarity quadrant at fixed recall, which is where HeadHunt-VAD's pooled features have no mechanism to help.
The saliency-masking competitor multiplies $z_{\text{stat}}$ by a generic attention map. We test two sources. One is the Attend to Anything human-attention model, run from its official code with a text prompt. The other is the discriminator-attention weighting of Kimura et al. This is the fairest test of whether a human-attention prior already supplies relevance. For the zero-shot semantic map we substitute StackCLIP's anomaly map, which reports 86.4 AUPRO on MVTec AD, into our combination rule in place of the head maps.
For numbers on COCO-AD, VisA-AC, and VIEW360 anomaly scores, we found no reported result on Lacuna for any baseline beyond InvAD-lite's 101.6 FPS speed and FDPN's relative gains. We therefore rerun all arms ourselves.
Three ablations isolate the key ingredient. The first uses HeadHunt-selected heads read as pooled features and fused late with the statistical map, removing spatiality. The second uses random heads and all-head averages, removing selection. The third fixes $c=0$, and separately uses a pure product, removing the interaction and additive terms. All learned components share the same calibration images and the same hyperparameter search budget.
Why not just…?
Why not just mask the statistical map with a saliency model? Masking can only lower scores, so it cannot recover anomalies that look statistically normal. Kimura et al. concede that it suppresses out-of-context background anomalies. The real threat is Attend to Anything conditioned on a text prompt, because where humans look plausibly correlates with where anomalies are. We treat this as a genuine rival. If AAM-weighting matches our method on the conforming-anomaly quadrant, the conclusion is that human attention priors suffice and the multimodal-head map is unnecessary, and we would report it that way.
Why not just ask the multimodal model, as VELM does? Generating text is slow, sensitive to prompt wording, and biased toward describing normal content, as HeadHunt-VAD documents. VELM also needs explicit defect definitions. The same-backbone VELM arm tests this directly. If it wins on accuracy, our remaining argument is cost and localization, and we would say so.
Why not just use HeadHunt-VAD's frame score next to a PatchCore map? This late fusion is precisely our first ablation. If spatial attention adds nothing beyond a global semantic score, late fusion will match us in the quadrant analysis. That outcome would falsify the core claim.
Why not just use CLIP zero-shot maps as the semantic axis? They may work, and StackCLIP is a strong map. Because the combination rule is agnostic to where the semantic map comes from, we test this by swapping the map source rather than arguing. Our hypothesis is that head maps from a language model are more robust to prompt wording than CLIP similarity maps. The paraphrase-stability measurement checks that hypothesis directly.
Decisive Checks
The experiments run on MVTec AD and MVTec-AC, VisA and VisA-AC, and COCO-AD for images, and on UCF-Crime and XD-Violence for video, with VIEW360 as a stress test for wide fields of view. We report image and pixel AUROC, AUPRO, and threshold-dependent mean F1, plus per-quadrant false alarm rate and recall at a fixed operating point, and throughput.
Three outcomes decide the proposal. First, on the benign-rarity quadrant, false alarms must fall substantially relative to PatchCore at unchanged recall. Second, on the conforming-anomaly quadrant, and especially COCO-AD's object intrusions, recall must rise. No masking baseline can do this, so it is the cleanest evidence that relevance carries information of its own. Third, the fitted $b$ and $c$ must be stable across calibration resamples, and spatial head maps must beat the late-fusion ablation. If the second outcome fails while the first succeeds, the method reduces to a better saliency mask. That would still be useful, but it would be a much smaller claim, and we would report it as such.
References
- HeadHunt-VAD: Hunting Robust Anomaly-Sensitive Heads in MLLM for Tuning-Free Video Anomaly Detection
- Transfer-Based Semantic Anomaly Detection
- Generative Cooperative Learning for Unsupervised Video Anomaly Detection
- Image Anomalies: A Review and Synthesis of Detection Methods
- Learning Feature Inversion for Multi-class Anomaly Detection under General-purpose COCO-AD Benchmark
- Adversarial Discriminative Attention for Robust Anomaly Detection
- Anomaly Detection for People with Visual Impairments using an Egocentric 360-Degree Camera
- Multilevel Saliency-Guided Self-Supervised Learning for Image Anomaly Detection
- Detect, Classify, Act: Categorizing Industrial Anomalies with Multi-Modal Large Language Models
- StackCLIP: Clustering-Driven Stacked Prompt in Zero-Shot Industrial Anomaly Detection
- ROADS: Robust Prompt-Driven Multi-Class Anomaly Detection Under Domain Shift
- Attend to Anything: Foundation Model for Unified Human Attention Modeling