TL;DR
This paper introduces PILOT, a novel framework for Zero-Shot Anomaly Detection (ZSAD) that tackles the challenge of domain shift. PILOT uses a dual-branch prompt learning mechanism and a label-free test-time adaptation strategy to achieve state-of-the-art performance in detecting and localizing anomalies in unseen categories, even when the test data distribution differs significantly from the training data.
What Problem Does This Paper Address?
Anomaly detection is crucial in safety-critical applications like industrial inspection and medical diagnostics. Traditional methods often require labeled anomaly data, which is scarce and expensive to obtain. Zero-Shot Anomaly Detection (ZSAD) aims to overcome this by leveraging pre-trained vision-language models (VLMs) like CLIP to identify anomalies without any target-domain annotations. However, existing ZSAD methods struggle with "domain shift" – performance degradation when the data distribution at test time differs from the training data. This is because their reliance on fixed or single learnable prompts, often trained on auxiliary datasets, limits their ability to generalize to new, unseen data distributions. The paper addresses this by proposing a method that can adapt to these domain shifts effectively using unlabeled test data.
Methods
PILOT introduces two key innovations:
- Dual-Branch Prompt Learning: Instead of a single prompt, PILOT uses a pool of learnable prompts and a fixed attribute memory bank.
- Learnable Prompt Pool (P): This pool contains multiple learnable prompts, each with associated embeddings for adaptive weighting and alignment. This allows the model to dynamically select and combine prompts that best match the input image's features.
- Attribute Memory Bank (U): This bank consists of fixed prompts derived from descriptive natural language templates (e.g., "normal object," "corroded object"). These provide stable semantic anchors, preventing overfitting and stabilizing adaptation.
- Prompt Aggregation: The representations from the learnable prompts and the attribute memory bank are fused using an orthogonal projection, which enriches the fused embedding with complementary information while anchoring it in the robust attribute space.
- Label-Free Test-Time Adaptation (TTA): During inference, PILOT adapts its learnable prompt parameters using unlabeled test data. It identifies high-confidence pseudo-labels from the test set and uses these to fine-tune the prompts, allowing the model to adjust to new domains without requiring any labeled target data. This adaptation is focused on the image-level objective to avoid issues with noisy pixel-level labels.
Datasets
The paper evaluates PILOT on 13 benchmark datasets spanning industrial inspection and medical imaging.
- Industrial Datasets (6): BTAD, DAGM, DTD-Synthetic, MVTec AD, SDD, VisA. These datasets feature diverse object categories and anomaly manifestations, including textured surfaces and object-centric scenarios with defects like cracks, scratches, and dents.
- Medical Datasets (7):
- Detection-only: BrainMRI, Br35H, HeadCT. These are used for binary anomaly classification on volumetric neuroimaging data.
- Localization: CVC-ClinicDB, CVC-ColonDB, ISIC, Kvasir. These datasets contain only abnormal samples and provide pixel-wise lesion annotations. TTA is not applied to these due to the lack of normal examples.
The paper uses MVTec AD and VisA as auxiliary datasets for training and evaluation on other datasets, respectively, to ensure strict zero-shot evaluation and demonstrate domain shift robustness.
Key Insights and Ideas
- Domain Shift is a Major Hurdle for ZSAD: The paper clearly demonstrates that existing ZSAD methods, even those using learnable prompts, struggle significantly when test data distributions differ from training data.
- A Pool of Prompts is Better Than One: Dynamically weighting and combining multiple learnable prompts, rather than relying on a single one, allows for more nuanced anomaly representation and better adaptation.
- Fixed Semantic Anchors Stabilize Adaptation: The attribute memory bank, with its fixed, semantically rich prompts, acts as a crucial anchor during TTA, preventing the learnable prompts from drifting and ensuring stable localization.
- Label-Free TTA is Powerful but Needs Care: While TTA can significantly improve performance, naive application can degrade results. PILOT's confidence-based pseudo-labeling and focused adaptation on learnable prompts prove effective.
- Image-Level TTA is More Robust: Focusing TTA on image-level objectives is shown to be more stable and less prone to noise than pixel-level adaptation, especially under domain shift.
Key Results
- State-of-the-Art Performance: PILOT achieves state-of-the-art results on 13 industrial and medical benchmarks for both image-level anomaly detection (AUROC, AP) and pixel-level localization (AUROC, AUPRO) in the zero-shot setting.
- Significant TTA Gains: The TTA strategy consistently improves performance across various datasets, particularly for image-level detection. For example, on MVTec AD, PILOT's image-level AUROC increases from 90.2 to 92.1 with TTA.
- Competitive with Full-Shot Methods: PILOT achieves detection accuracy on par with full-shot methods (which have access to target domain data) while operating in a zero-shot setting.
- Efficient Adaptation: PILOT's TTA phase only updates a small subset of parameters (learnable prompts), making it computationally efficient.
Limitations and Future Work
- TTA Focus on Image-Level: While effective, the current TTA strategy focuses on image-level objectives. The authors suggest future work could explore advanced TTA strategies to jointly improve pixel-level localization and image-level detection for even greater ZSAD robustness.
- Medical Datasets Lack Normal Samples: For medical localization datasets, TTA could not be applied due to the absence of normal examples. This limits the evaluation of PILOT's full capabilities in these domains.
- Potential for Further Improvement in Localization: While PILOT shows strong localization performance, the paper notes that there is still room for improvement, especially in complex scenarios.
Summary updated