Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection

Jiawen Zhu, Yew-Soon Ong, Chunhua Shen, Guansong Pang

Abstract

Current zero-shot anomaly detection (ZSAD) methods show remarkable success in prompting large pre-trained vision-language models to detect anomalies in a target dataset without using any dataset-specific training or demonstration. However, these methods often focus on crafting/learning prompts that capture only coarse-grained semantics of abnormality, e.g., high-level semantics like "damaged", "imperfect", or "defective" objects. They therefore have limited capability in recognizing diverse abnormality details that deviate from these general abnormal patterns in various ways. To address this limitation, we propose FAPrompt, a novel framework designed to learn Fine-grained Abnormality Prompts for accurate ZSAD. To this end, a novel Compound Abnormality Prompt learning (CAP) module is introduced in FAPrompt to learn a set of complementary, decomposed abnormality prompts, where abnormality prompts are enforced to model diverse abnormal patterns derived from the same normality semantic. On the other hand, the fine-grained abnormality patterns can be different from one dataset to another. To enhance the cross-dataset generalization, another novel module, namely Data-dependent Abnormality Prior learning (DAP), is introduced in FAPrompt to learn a sample-wise abnormality prior from abnormal features of each test image to dynamically adapt the abnormality prompts to individual test images. Comprehensive experiments on 19 real-world datasets, covering both industrial defects and medical anomalies, demonstrate that FAPrompt substantially outperforms state-of-the-art methods in both image- and pixel-level ZSAD tasks.

Addressing the "Granularity Gap"

Zero-shot Anomaly Detection (ZSAD) represents a significant challenge in industrial inspection and medical imaging. The objective is to identify defects or pathologies in novel datasets without any domain-specific training. While current state-of-the-art methods utilize Vision-Language Models (VLMs) like CLIP by comparing image features to generic prompts (e.g., "normal" vs. "damaged"), they suffer from a "granularity gap." Real-world anomalies are highly diverse; a "damaged" item could manifest as a hole, a stain, or a structural crack. Static, coarse-grained prompts fail to capture these subtle variations.

The FAPrompt Framework

To overcome these limitations, the authors introduce FAPrompt, a framework built on two key innovations:

  1. Compound Abnormality Prompt (CAP) Learning: Instead of a single generic prompt, CAP learns $K$ complementary prompts. These are "compounded" by sharing base "normal" tokens while adding unique learnable "abnormal" tokens. An orthogonal constraint loss ($L_{oc}$) ensures these prompts capture distinct, non-overlapping aspects of abnormality.
  2. Data-dependent Abnormality Prior (DAP) Learning: This module introduces dynamic adaptation. For each test image, DAP identifies the top $M$ "suspicious" patches and extracts features to create a sample-wise "abnormality prior." This prior "tunes" the CAP prompts based on the specific image content. A prior learning loss ($L_{prior}$) is used to suppress noise from normal images.
Overview of the FAPrompt framework, featuring the CAP and DAP modules.

The model is trained on an auxiliary dataset (e.g., MVTecAD) to learn these structures and then generalizes to target datasets without further tuning.

Experimental Evaluation

The researchers evaluated FAPrompt across 19 real-world datasets, spanning industrial defects (e.g., VisA, BTAD, SDD) and medical anomalies (e.g., BrainMRI, HeadCT, ISIC). This extensive testing demonstrates the framework's robustness across diverse visual domains.

Key Results

FAPrompt consistently outperforms existing leaders like AnomalyCLIP and FiLO:

  • Performance Gains: It achieves an average improvement of 3% to 5% in both image-level detection and pixel-level localization.
  • Industrial & Medical Success: Notable results include 92.2% AUROC on the BTAD dataset and 95.8% AUROC on BrainMRI.
  • Visualization: t-SNE analysis confirms that FAPrompt’s learned prompts are more distinct and spread out than those of previous methods, effectively capturing a wider spectrum of abnormal patterns.
Comparison of prompt learning strategies and t-SNE visualizations of learned abnormality prompts.

Insights and Future Directions

FAPrompt shifts the paradigm from treating anomalies as broad categories to viewing them as specific deviations from a known norm. The use of orthogonal constraints effectively simulates various anomaly "classes" without explicit labels, while the DAP module allows the model to "zoom in" on suspicious features dynamically.

However, some limitations remain. The model's performance can be sensitive to the choice of auxiliary training data and hyperparameters like the number of prompts ($K$) and patches ($M$). Additionally, the increased complexity of multiple prompts and dynamic prior networks may impact real-time deployment on edge devices. Future work may focus on optimizing these aspects for broader industrial application.

Code is available at https://github.com/mala-lab/FAPrompt.

Summary updated

Abstract

Current zero-shot anomaly detection (ZSAD) methods show remarkable success in prompting large pre-trained vision-language models to detect anomalies in a target dataset without using any dataset-specific training or demonstration. However, these methods often focus on crafting/learning prompts that capture only coarse-grained semantics of abnormality, e.g., high-level semantics like "damaged", "imperfect", or "defective" objects. They therefore have limited capability in recognizing diverse abnormality details that deviate from these general abnormal patterns in various ways. To address this limitation, we propose FAPrompt, a novel framework designed to learn Fine-grained Abnormality Prompts for accurate ZSAD. To this end, a novel Compound Abnormality Prompt learning (CAP) module is introduced in FAPrompt to learn a set of complementary, decomposed abnormality prompts, where abnormality prompts are enforced to model diverse abnormal patterns derived from the same normality semantic. On the other hand, the fine-grained abnormality patterns can be different from one dataset to another. To enhance the cross-dataset generalization, another novel module, namely Data-dependent Abnormality Prior learning (DAP), is introduced in FAPrompt to learn a sample-wise abnormality prior from abnormal features of each test image to dynamically adapt the abnormality prompts to individual test images. Comprehensive experiments on 19 real-world datasets, covering both industrial defects and medical anomalies, demonstrate that FAPrompt substantially outperforms state-of-the-art methods in both image- and pixel-level ZSAD tasks. Code is available at https://github.com/mala-lab/FAPrompt.
BibTeX
@misc{zhu2024fine,
  author = {Jiawen Zhu and Yew-Soon Ong and Chunhua Shen and Guansong Pang},
  title = {{Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection}},
  year = {2024},
  eprint = {2410.10289},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.10289},
}
Figure