PromptAD: Zero-shot Anomaly Detection using Text Prompts

Yiting Li, Adam David Goodge, Fayao Liu, Chuan-Sheng Foo

Introduction

Traditional Anomaly Detection (AD) models are typically specialists, requiring extensive "normal" data for a specific object category to define what constitutes a defect. Consequently, a model trained to identify defects in one category, such as a "hazelnut," often fails when applied to a different category, like a "metal nut" or a "capsule." This limitation necessitates retraining for every new application, which is costly and time-consuming.

PromptAD addresses this by introducing a Zero-Shot Anomaly Detection (ZSAD) framework. By leveraging the multi-modal capabilities of CLIP, PromptAD uses natural language prompts to describe both "normal" and "abnormal" states, allowing it to generalize to entirely new object categories without additional training or visual examples.

The PromptAD Framework

PromptAD is built upon a frozen CLIP backbone and utilizes a dual-branch architecture to bridge the gap between visual information and semantic descriptions.

1. The Normality Branch

This branch establishes a "profile of normality" by using text prompts to define what a healthy object should look like. For instance, a prompt might describe "a short metal pin with a helical thread."

A normal hazelnut representing the baseline for the normality branch.

2. The Abnormality Branch

In contrast, the abnormality branch explicitly models anomalous behaviors using prompts like "crack," "scratch," or "hole." This allows the model to learn specific defect patterns that can be applied across different object types.

An anomalous hazelnut with a hole, illustrating the target for the abnormality branch.

Enhancing Generalization through Interaction

To ensure these branches work synergistically and avoid overfitting to training classes, PromptAD introduces two key interaction components:

  • Cross-view Contrastive Learning (CCL): This regularizes training by using information from one branch to refine the other, calculating residuals to ensure the model learns transferable features.
  • Cross-view Mutual Interaction (CMI): This acts as a feedback loop. If the normality branch detects a deviation missed by the abnormality branch, it provides an attention map to guide the abnormality branch toward the neglected region.

The model is optimized using a Symmetric Deviation Loss, which enforces a statistically significant gap between the scores assigned to normal and anomalous pixels.

Experimental Results

The researchers evaluated PromptAD on major industrial benchmarks, including MVTec AD and AITEX, using a leave-one-category-out setup to test zero-shot generalization.

PromptAD significantly outperformed existing state-of-the-art baselines:

  • Classification: On MVTec AD (1-shot), PromptAD achieved an average AUC of 0.908, compared to 0.815 for the previous best-performing method (DRA).
  • Localization: For pixel-level anomaly localization, PromptAD reached a pAUROC of 0.921, surpassing WinCLIP (0.851).
  • Efficiency: The model demonstrated high "data efficiency," where well-designed text prompts effectively replaced the need for hundreds of labeled images.

Conclusion

PromptAD demonstrates that high-quality text descriptions can compensate for missing visual data in anomaly detection. By modeling both normality and abnormality and facilitating their interaction, the framework achieves superior zero-shot performance. Future work will focus on automating prompt generation and improving robustness to complex, non-industrial backgrounds.

Summary updated

Abstract

We consider the problem of zero-shot anomaly detection in which a model is pre-trained to detect anomalies in images belonging to seen classes, and expected to detect anomalies from unseen classes at test time. State-of-the-art anomaly detection (AD) methods can often achieve exceptional results when training images are abundant, but they catastrophically fail in zero-shot scenarios with a lack of real examples. However, with the emergence of multi-modal models such as CLIP, it is possible to use knowledge from other modalities (e.g. text) to compensate for the lack of visual information and improve AD performance. In this work, we propose PromptAD, a dual-branch framework which uses prior knowledge about both normal and abnormal behaviours in the form of text prompts to detect anomalies even in unseen classes. More specifically, it uses CLIP as a backbone encoder network and an additional dual-branch vision-language decoding network for both normality and abnormality information. The normality branch establishes a profile of normality, while the abnormality branch models anomalous behaviors, guided by natural language text prompts. As the two branches capture complementary information or ‘views’, we propose a ‘cross-view contrastive learning’ (CCL) component which regularizes each view with additional reference information from the other view. We further propose a cross-view mutual interaction (CMI) strategy to promote the mutual exploration of useful knowledge from each branch. We show that PromptAD outperforms existing baselines in zero-shot anomaly detection on key benchmark datasets and analyse the role of each component in ablation studies.
BibTeX
@inproceedings{li2024promptad,
  author = {Yiting Li and Adam David Goodge and Fayao Liu and Chuan-Sheng Foo},
  title = {{PromptAD: Zero-shot Anomaly Detection using Text Prompts}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00113},
}
Figure