TL;DR
Existing zero-shot anomaly detection (ZSAD) methods rely on coarse, high-level abnormality prompts (like "damaged" or "defective"), which fail to recognize diverse, subtle defects. To solve this, FAPrompt introduces a compound prompt design constrained to learn diverse deviations from normality, paired with a sample-specific abnormality prior derived dynamically from test images. This setup sets a new state-of-the-art across 19 diverse industrial and medical datasets, boosting image- and pixel-level anomaly detection by 3%–5%.
What Problem Does This Paper Address?
Vision-Language Models (VLMs) like CLIP have enabled impressive zero-shot anomaly detection, where a model detects visual defects in unseen datasets without task-specific training. However, standard methods either use generic handcrafted prompts or learn single, coarse abnormal tokens.
In practice, anomalies rarely present as completely new semantic classes. Instead, they appear as subtle, localized deviations from normal patterns (e.g., small color stains or tiny cracks on a textured surface). Relying on broad descriptors like "damaged object" causes models to learn overly simple, homogeneous abnormal representations that generalize poorly across domains and fail on diverse defect types.
Methods
FAPrompt freezes the pre-trained CLIP visual and text backbones and introduces two core modules:
[ Normal Prompt: V_1 ... V_E + "object" ]
│
▼ (compounded with abnormal tokens A_j)
[ Fine-Grained Abnormality Prompts: P^{a_1}, ..., P^{a_K} ] ──► Orthogonal Loss (L_oc)
▲
│ (conditioned via addition)
[ Abnormality Prior Ω_x ] ◄── Learned from Top-M patch tokens of image x
- Compound Abnormality Prompt Learning (CAP):
- Instead of learning normal and abnormal tokens independently, each abnormality prompt compounds the shared learnable normal tokens with a few specialized abnormal tokens ($\mathcal{P}^{a_i} = [V_1 \dots V_E][A_1^i \dots A_{E'}^i][\text{object}]$). This ensures abnormal prompts stay grounded in the baseline concept of normality while capturing subtle deviations.
- An orthogonal constraint loss ($\mathcal{L}_{oc}$) enforces that the learned abnormality prompt embeddings remain mutually diverse and non-redundant. The individual prompts are then averaged into a single prototype embedding ($\mathbf{F}_a$).
- Data-dependent Abnormality Prior Learning (DAP):
- Fine-grained abnormalities vary drastically between datasets. To ground the prompts dynamically, DAP identifies the top $M$ most abnormal patch tokens from the query image and passes them through an abnormality prior network $\psi(\cdot)$ to produce a vector $\Omega_x$.
- $\Omega_x$ is added directly to the abnormal tokens before text encoding.
- To prevent clean/normal test images from corrupting prompts with noise, a regularizing prior loss ($\mathcal{L}_{prior}$) forces $\Omega_x$ toward a null vector whenever the input image is normal.
Datasets
The authors evaluated FAPrompt across 19 real-world datasets (models were trained on MVTecAD and evaluated on the rest without fine-tuning, or trained on VisA when evaluating MVTecAD):
- 9 Industrial Defect Datasets: MVTecAD, VisA, BTAD, DAGM, DTD-Synthetic, AITEX, SDD, MPDD, and ELPV.
- 10 Medical Anomaly Datasets: BrainMRI, HeadCT, LAG, Br35H (image-level evaluation), along with CVC-ColonDB, CVC-ClinicDB, Kvasir, Endo, ISIC, and TN3K (pixel-level evaluation).
- No new datasets were introduced; the study relies entirely on public defect and pathology benchmarks.
Key Insights and Ideas
- Abnormalities are variations of normality, not separate categories: Compounding abnormal tokens directly onto the normal token sequence prevents prompt drift and anchors the defect search to contextual deviations.
- Ensembles don't solve prompt collapse: Simply learning multiple prompts with different random seeds (or multi-model ensembles) produces highly redundant features. FAPrompt’s explicit orthogonality constraint ($\mathcal{L}_{oc}$) forces the model to decompose the defect space into genuinely complementary sub-patterns.
- Image-level priors are too noisy for anomaly detection: Methods like CoCoOp condition prompts on global image tokens. For anomaly detection, global image tokens introduce background noise—especially on normal images. Extracting priors strictly from top anomalous patches, combined with a suppression loss on normal samples, is critical for effective conditioning.
- Simple averaging beats selection: Averaging multiple diverse abnormal prompts into a single prototype consistently outperformed dynamically selecting the "best-matched" prompt or learning weighted combinations, which were prone to overfitting on the auxiliary dataset.
Key Results
- Image-Level ZSAD: FAPrompt achieved an average improvement of up to 3.9% AUROC / 3.0% AP on industrial benchmarks and 2.3% AUROC / 1.3% AP on medical benchmarks over prior SotA models (e.g., AnomalyCLIP, FiLO).
- Pixel-Level Localization: Outperformed prior methods by up to 1.4% AUROC / 4.0% PRO on industrial datasets and 3.2% AUROC / 4.2% PRO on medical datasets.
- Hyperparameter Robustness: Optimal performance on industrial datasets occurred around $K = 10$ abnormality prompts and $M = 10$ selected abnormal patch candidates.
Limitations and Future Work
- Reliance on Patch Heuristics: The DAP module relies on an initial similarity ranking to identify the top $M$ abnormal patches. In cases where anomalies are subtle, diffuse, or low-contrast, the initial pass may select uninformative patches.
- Sensitivity to Large $K$ and $M$: Increasing the number of prompts ($K > 10$) or selected patches ($M > 10$) causes slight performance degradation due to redundancy and noise.
- Domain Specialization: While cross-dataset transfer to medical benchmarks showed strong improvements, performance gaps remain between defect-centric pretraining and nuanced medical pathologies where normal morphology is highly variable.
Summary updated