Current zero-shot anomaly detectors can tell an inspector or clinician that something is wrong in a new product line or organ they were never trained on. They cannot say what is wrong. A useful system has to answer "where is the defect, and which of ten thousand catalogued failure modes is it?" from a single forward pass, fast enough for a live camera or endoscope, and without anyone tracing masks for each failure mode. This proposal builds that system.
Its central bet is that the right thing to name is the deviation, not the object. The method first finds anomalous pixels cheaply. It then pools them into a few region vectors and subtracts the image's own estimate of what that region should look like. Only this difference vector, the residual, is matched against text descriptions of every label in the taxonomy. Because naming runs on a handful of residuals instead of every pixel, and the residual carries "scratched" rather than "screw," the method should be both faster and more accurate than cropping suspicious regions and asking a vision-language model what it sees.
Core Claim
Zero-shot visual anomaly detection is now fast and accurate as a binary problem. UniADet learns a tiny head of about 2,000 parameters on an auxiliary dataset (VisA). That head separates image-level from pixel-level decision weights, and with it UniADet reaches 82.5% image-level AUROC on Real-IAD at 15.7 ms per image. Its output is still a single normal/abnormal score. Multi-label open-vocabulary recognition has a different weakness. TagCLIP fixes the tendency of CLIP (a contrastively trained image–text model) to see only the most prominent object by classifying patches from its penultimate layer. It then re-crops each candidate and re-encodes it, which costs one extra backbone pass per candidate class.
We claim three things. First, anomaly detection should act as the gate that decides where label scoring happens at all. Second, naming should be decoupled from localization, extending UniADet's finding that the two objectives conflict when they share weights. Third, the naming head should score residuals: a region's feature minus an intrinsic normal prototype extracted from the same image, in the spirit of INP-Former. Concretely, for a detected region $r$ with pooled feature $z_r$ and intrinsic normal prototype $p_r$, the score of label $j$ is
$$ s_j(r) = \big\langle A\,(z_r - p_r),\; t_j - t_{\mathrm{norm}(j)} \big\rangle , $$
where $t_j$ is the text embedding of label $j$'s description, $t_{\mathrm{norm}(j)}$ is the embedding of the matching normal description (for example "an intact metal screw" for "a cross-threaded metal screw"), and $A$ is a single learned linear map from visual residuals to text differences. Scoring costs $R \cdot L \cdot d$ multiply-adds for $R$ regions, $L$ labels and embedding width $d$. With $R \le 8$, $L = 10^5$ and $d = 768$, this is a single small matrix product, so the label space never touches the per-pixel path.
Research Program
The system has four parts. A frozen DINOv2 backbone runs once per frame. A UniADet-style decoupled detection head produces an anomaly map and an image score. An INP-style extractor provides a small set of prototype tokens summarizing the image's normal appearance. The naming head turns the thresholded anomaly map into at most $R$ connected regions, pools backbone features inside each region, subtracts the nearest prototype, and scores the result against the label bank with the equation above. The label bank is built offline once: every label gets a description and a paired normal description, each encoded by a frozen text encoder. Adding labels later requires no retraining.
The annotation bottleneck shapes how $A$ is trained. We use no masks and no region labels. The only supervision is image-level defect-type tags, which come almost for free: MVTec-AD, VisA and Real-IAD already organize anomalous images by defect type, and medical archives carry report-level diagnoses. Training is multi-instance. An image's tag must be explained by at least one of its regions, and no region may claim a label the image does not carry. This borrows the bag-level consistency idea behind the max-severity constraint of Yamaguchi et al., which beat semi-supervised training with 5% frame labels using only patient-level diagnoses. As in UniADet, $A$ is trained on auxiliary categories and evaluated on held-out object categories and held-out defect labels. Both kinds of generalization are zero-shot.
The label structure question is left open on purpose. Probabilistic Label Trees give logarithmic inference and provably consistent multi-label ranking, and they show that greedy hierarchical softmax is inconsistent. At region level, however, flat scoring is already cheap. A tree therefore earns its place only if routing through parent nodes ("surface defect" → "scratch" → "fine linear scratch on brushed metal") improves precision on near-duplicate labels. We test this rather than assume it. HASTE is not used. Its grouped sparse output layer requires supervised training on every label, which is exactly what we lack.
For video, the system is per-frame with one cheap addition: a running label distribution per tracked region. That distribution is updated only when consecutive frames agree, a simplified version of the continuity gate in VPSentry. Temporal modeling is a robustness add-on here, not the contribution. One limitation is inherited from INP-Former, which reports that it fails on logical anomalies where every patch looks locally healthy. Residual naming cannot name what the detector never flags, and we will report that failure explicitly.
Baselines
The naive reference point is flat CLIP prompt scoring. Every patch's CLIP embedding is compared with "a photo of a [label] [object]" for all $L$ labels, and the image is assigned the top-scoring labels. This is what a practitioner would try first, and it exposes both the per-patch $O(NL)$ cost and the dominant-object bias that TagCLIP documents. TagCLIP reports vanilla CLIP at 85.8% versus 92.8% mAP on PASCAL VOC 2007 for multi-label recognition.
The strongest competitor is a detect-then-crop pipeline, which we assemble ourselves because no single paper does it. UniADet produces the anomaly map. Each detected region is cropped and re-encoded by CLIP, exactly as in TagCLIP's class-wise reidentification, and the crop is scored against the same label bank and descriptions. This pipeline shares our detector, our regions and our text, so it isolates the one thing we change: naming the residual instead of re-encoding the raw crop. TagCLIP's reported multi-label figures are 92.8% mAP on VOC 2007 and 68.8% on COCO, but no reported result was found on Lacuna for defect-type naming on MVTec-AD, VisA or Real-IAD by any method. Our benchmark numbers for this baseline will therefore be the first. For detection itself, the bar is UniADet's reported 82.5% image-level AUROC on Real-IAD and 15.7 ms latency. Our system must not lose detection accuracy and must stay under 33 ms per frame on the same GPU. Beating the strongest baseline means equal or higher top-1 and Precision@5 naming accuracy on held-out defect labels at lower latency, sustained as $L$ grows.
Every method gets the same backbones, region proposals, label descriptions and auxiliary training data. Learned components get the same hyperparameter search budget, and all methods are timed with identical batching on one GPU model plus one low-power edge card.
Three ablations isolate the key ingredient. The first replaces $z_r - p_r$ with $z_r$ alone, keeping $A$, so it tests whether subtracting the prototype matters. The second replaces the text difference $t_j - t_{\mathrm{norm}(j)}$ with $t_j$ alone. The third swaps flat scoring for a probabilistic label tree built on the label-description embeddings.
Why not just…?
Why not just crop each suspicious region and ask CLIP what it is? This is the real threat, and it is the strongest baseline above. Its cost grows with the number of regions, one backbone pass each, which TagCLIP's design makes explicit. The deeper concern is accuracy. A tight crop of a scratched screw is still mostly screw, so CLIP's embedding will be dominated by object identity, and fine defect labels will tie. If the crop pipeline matches our accuracy on held-out defect labels, the residual idea is wrong. In that case the contribution shrinks to an engineering result about batching crops, and we will report it as such.
Why not just score every patch against every label and skip detection? Accuracy is part of the problem: patch-level CLIP scoring inherits the dominant-object bias TagCLIP measured. Cost is the rest. With roughly a thousand patches and $10^5$ labels, the per-frame product is about a hundred times larger than region-level scoring and grows with resolution. The label-scaling experiment below plots latency and Precision@5 against $L$ for this baseline to show where it breaks.
Why not just train a supervised defect classifier on the auxiliary datasets? Such a classifier would likely win on the defect types it was trained on. It cannot name labels absent from training, and those are precisely the long-tail failure modes that motivate the work. Our split holds out entire defect labels, so this baseline scores zero on them by construction. We still report it on seen labels to show how much accuracy the zero-shot design gives up there.
Why not just caption the region with a language model and retrieve the label, as ICXML does for text? ICXML generates candidate labels with a language model, retrieves matching taxonomy entries, then has the model rerank them. On LF-WikiSeeAlso-320K this reaches 0.278 Precision@1 against 0.237 for retrieval alone, but it takes seconds per instance, two orders of magnitude over a video frame budget. We include a single offline run of this pipeline as an accuracy ceiling, not as a deployable competitor.
Proposed Experiments
The primary benchmark merges MVTec-AD, VisA and Real-IAD into one taxonomy of object–defect labels, each with written descriptions. $A$ is trained on VisA categories and defect types. Evaluation covers MVTec-AD and Real-IAD categories, which are unseen objects, and held-out defect types, which are unseen labels. Metrics are top-1 accuracy and Precision@5 over anomalous regions, together with UniADet's image AUROC and pixel average precision to confirm detection is unharmed.
To test scale, we pad the label bank with $10^3$ to $10^5$ distractor descriptions. These are built by recombining materials, objects and defect verbs and are deliberately hard to tell apart from true labels. For every method we plot naming accuracy and per-frame latency against $L$. This padded taxonomy is synthetic, and we say so. It measures robustness to confusable vocabulary, not real clinical coding.
The medical transfer test uses image-level diagnostic categories in the medical benchmarks that UniADet evaluates on, such as ISIC. Training uses only report-level tags.
The video test runs the full system on SUN-SEG clips. Detection quality is compared against VPSentry's reported 83.95 Dice on SUN-SEG-Hard under 10% supervision, keeping in mind that our system uses no polyp masks at all. We also measure label stability across frames with and without the continuity gate.
The decisive check is the first ablation. If residual naming does not beat raw-region naming by a clear margin on held-out defect labels, and if that margin does not widen as distractor labels are added, the central claim fails regardless of latency.
References
- One Language-Free Foundation Model Is Enough for Universal Vision Anomaly Detection
- TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP without Training
- Exploring Intrinsic Normal Prototypes within a Single Image for Universal Anomaly Detection
- Domain Adaptation for Ulcerative Colitis Severity Estimation Using Patient-Level Diagnoses
- Probabilistic Label Trees for Extreme Multi-label Classification
- HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spaces
- VPSentry: Semi-supervised Video Polyp Segmentation via Sentry-guided Long-term Prototype Fusion with Correlation Dynamic Propagation
- ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classification