Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection

Jiawen Zhu, Yew-Soon Ong, Chunhua Shen, Guansong PangSubmission to ICLR 2025 · Rejected

TL;DR

Existing zero-shot anomaly detection (ZSAD) methods rely on coarse, high-level abnormality prompts (like "damaged" or "defective"), which fail to recognize diverse, subtle defects. To solve this, FAPrompt introduces a compound prompt design constrained to learn diverse deviations from normality, paired with a sample-specific abnormality prior derived dynamically from test images. This setup sets a new state-of-the-art across 19 diverse industrial and medical datasets, boosting image- and pixel-level anomaly detection by 3%–5%.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) like CLIP have enabled impressive zero-shot anomaly detection, where a model detects visual defects in unseen datasets without task-specific training. However, standard methods either use generic handcrafted prompts or learn single, coarse abnormal tokens.

In practice, anomalies rarely present as completely new semantic classes. Instead, they appear as subtle, localized deviations from normal patterns (e.g., small color stains or tiny cracks on a textured surface). Relying on broad descriptors like "damaged object" causes models to learn overly simple, homogeneous abnormal representations that generalize poorly across domains and fail on diverse defect types.

Methods

FAPrompt freezes the pre-trained CLIP visual and text backbones and introduces two core modules:

[ Normal Prompt: V_1 ... V_E + "object" ]
                  │
                  ▼ (compounded with abnormal tokens A_j)
[ Fine-Grained Abnormality Prompts: P^{a_1}, ..., P^{a_K} ] ──► Orthogonal Loss (L_oc)
                  ▲
                  │ (conditioned via addition)
[ Abnormality Prior Ω_x ] ◄── Learned from Top-M patch tokens of image x
  1. Compound Abnormality Prompt Learning (CAP):
  2. Instead of learning normal and abnormal tokens independently, each abnormality prompt compounds the shared learnable normal tokens with a few specialized abnormal tokens ($\mathcal{P}^{a_i} = [V_1 \dots V_E][A_1^i \dots A_{E'}^i][\text{object}]$). This ensures abnormal prompts stay grounded in the baseline concept of normality while capturing subtle deviations.
  3. An orthogonal constraint loss ($\mathcal{L}_{oc}$) enforces that the learned abnormality prompt embeddings remain mutually diverse and non-redundant. The individual prompts are then averaged into a single prototype embedding ($\mathbf{F}_a$).
  4. Data-dependent Abnormality Prior Learning (DAP):
  5. Fine-grained abnormalities vary drastically between datasets. To ground the prompts dynamically, DAP identifies the top $M$ most abnormal patch tokens from the query image and passes them through an abnormality prior network $\psi(\cdot)$ to produce a vector $\Omega_x$.
  6. $\Omega_x$ is added directly to the abnormal tokens before text encoding.
  7. To prevent clean/normal test images from corrupting prompts with noise, a regularizing prior loss ($\mathcal{L}_{prior}$) forces $\Omega_x$ toward a null vector whenever the input image is normal.

Datasets

The authors evaluated FAPrompt across 19 real-world datasets (models were trained on MVTecAD and evaluated on the rest without fine-tuning, or trained on VisA when evaluating MVTecAD):

  • 9 Industrial Defect Datasets: MVTecAD, VisA, BTAD, DAGM, DTD-Synthetic, AITEX, SDD, MPDD, and ELPV.
  • 10 Medical Anomaly Datasets: BrainMRI, HeadCT, LAG, Br35H (image-level evaluation), along with CVC-ColonDB, CVC-ClinicDB, Kvasir, Endo, ISIC, and TN3K (pixel-level evaluation).
  • No new datasets were introduced; the study relies entirely on public defect and pathology benchmarks.

Key Insights and Ideas

  • Abnormalities are variations of normality, not separate categories: Compounding abnormal tokens directly onto the normal token sequence prevents prompt drift and anchors the defect search to contextual deviations.
  • Ensembles don't solve prompt collapse: Simply learning multiple prompts with different random seeds (or multi-model ensembles) produces highly redundant features. FAPrompt’s explicit orthogonality constraint ($\mathcal{L}_{oc}$) forces the model to decompose the defect space into genuinely complementary sub-patterns.
  • Image-level priors are too noisy for anomaly detection: Methods like CoCoOp condition prompts on global image tokens. For anomaly detection, global image tokens introduce background noise—especially on normal images. Extracting priors strictly from top anomalous patches, combined with a suppression loss on normal samples, is critical for effective conditioning.
  • Simple averaging beats selection: Averaging multiple diverse abnormal prompts into a single prototype consistently outperformed dynamically selecting the "best-matched" prompt or learning weighted combinations, which were prone to overfitting on the auxiliary dataset.

Key Results

  • Image-Level ZSAD: FAPrompt achieved an average improvement of up to 3.9% AUROC / 3.0% AP on industrial benchmarks and 2.3% AUROC / 1.3% AP on medical benchmarks over prior SotA models (e.g., AnomalyCLIP, FiLO).
  • Pixel-Level Localization: Outperformed prior methods by up to 1.4% AUROC / 4.0% PRO on industrial datasets and 3.2% AUROC / 4.2% PRO on medical datasets.
  • Hyperparameter Robustness: Optimal performance on industrial datasets occurred around $K = 10$ abnormality prompts and $M = 10$ selected abnormal patch candidates.

Limitations and Future Work

  • Reliance on Patch Heuristics: The DAP module relies on an initial similarity ranking to identify the top $M$ abnormal patches. In cases where anomalies are subtle, diffuse, or low-contrast, the initial pass may select uninformative patches.
  • Sensitivity to Large $K$ and $M$: Increasing the number of prompts ($K > 10$) or selected patches ($M > 10$) causes slight performance degradation due to redundancy and noise.
  • Domain Specialization: While cross-dataset transfer to medical benchmarks showed strong improvements, performance gaps remain between defect-centric pretraining and nuanced medical pathologies where normal morphology is highly variable.

Summary updated

Abstract

Current zero-shot anomaly detection (ZSAD) methods show remarkable success in prompting large pre-trained vision-language models to detect anomalies in a target dataset without using any dataset-specific training or demonstration. However, these methods are often focused on crafting/learning prompts that capture only coarse-grained semantics of abnormality, $e.g.$, high-level semantics like "damaged", "imperfect", or "defective" on carpet. They therefore have limited capability in recognizing diverse abnormality details that deviate from these general abnormal patterns in various ways. To address this limitation, we propose FAPrompt, a novel framework designed to learn Fine-grained Abnormality Prompts for more accurate ZSAD. To this end, we introduce a novel compound abnormality prompting module in FAPrompt to learn a set of complementary, decomposed abnormality prompts, where each abnormality prompt is formed by a compound of shared normal tokens and a few learnable abnormal tokens. On the other hand, the fine-grained abnormality patterns can be very different from one dataset to another. To enhance their cross-dataset generalization, we further introduce a data-dependent abnormality prior module that learns to derive abnormality features from each query/test image as a sample-wise abnormality prior to ground the abnormality prompts in a given target dataset. Comprehensive experiments conducted across 19 real-world datasets, covering both industrial defects and medical anomalies, demonstrate that FAPrompt substantially outperforms state-of-the-art methods by at least 3%-5% AUC/AP in both image- and pixel-level ZSAD tasks.
BibTeX
@misc{zhu2024fine,
  author = {JIAWEN ZHU and Yew-Soon Ong and Chunhua Shen and Guansong Pang},
  title = {{Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection}},
  year = {2024},
  eprint = {2410.10289},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.10289},
}
Figure