KAnoCLIP: Zero-Shot Anomaly Detection through Knowledge-Driven Prompt Learning and Enhanced Cross-Modal Integration

Chengyuan Li, Suyang Zhou, Jieping Kong, Lei Qi, Hui XueICASSP 2025 · DOI

TL;DR

This paper introduces KAnoCLIP, a novel zero-shot anomaly detection framework that leverages large language models (LLMs) and visual question answering (VQA) systems to generate learnable anomaly prompts. This approach eliminates the need for manually crafted prompts, significantly improving generalization and enabling more precise pixel-level anomaly segmentation.

What Problem Does This Paper Address?

Anomaly detection (AD) is critical in various fields, but collecting labeled data for every possible anomaly is often impractical due to cost, time, and privacy concerns. Traditional methods struggle with the diverse nature of anomalies. Zero-shot anomaly detection (ZSAD) aims to identify anomalies without specific training examples. While vision-language models like CLIP have shown promise, they face two main challenges: 1) relying on fixed, manually crafted textual descriptions for anomalies is time-consuming, prone to ambiguity, and limits generalization. 2) CLIP's focus on global semantics hinders its ability to perform precise pixel-level anomaly segmentation. This paper addresses these limitations by developing a more robust and generalizable ZSAD framework.

Methods

KAnoCLIP introduces Knowledge-Driven Prompt Learning (KnPL), a core innovation that addresses the limitations of fixed prompts. KnPL leverages an LLM (GPT-3.5) for general anomaly knowledge and a VQA system (Llama3) for image-specific anomaly descriptions. This forms an LLM-VQA knowledge base, which guides the generation of learnable normal prompts (LNPs) and learnable abnormal prompts (LAPs) through a novel Knowledge-Driven (KD) loss function. This loss function minimizes the distance between generated abnormal prompts and LAPs while maximizing the distance to LNPs, effectively removing the reliance on manual prompts and improving generalization.

Beyond prompt learning, KAnoCLIP incorporates three key modules:

  • CLIP-VV (Visual Encoder with V-V Attention): This module enhances the CLIP visual encoder by incorporating V-V attention. This mechanism refines the extraction of local visual features, crucial for fine-grained anomaly detection, by focusing on relationships between local features without disrupting the original CLIP structure.
  • Bi-Directional Cross-Attention for Multi-Level Cross-Modal Interaction (Bi-CMCI): This module addresses limitations in traditional cross-modal fusion. It uses bi-directional cross-attention (BiCA) to align textual descriptions with visual parts (Text-to-Visual) and refine text with local visual features (Visual-to-Text). This allows for capturing complex interactions between visual and textual modalities across multiple stages of the visual encoder.
  • Conv-Adapter: This module further refines global visual features by aligning them with textual information. It uses a depthwise separable convolution bottleneck to capture spatial features and reduce dimensionality, improving the integration of global context.

These components are jointly optimized using a total loss function that combines the KD loss, a global loss (binary cross-entropy), and a local loss (focal and dice loss) for pixel-level anomaly detection.

Datasets

The paper evaluates KAnoCLIP on a total of 12 datasets, encompassing both industrial and medical domains:

  • Industrial Datasets: MVTec-AD, VisA, MPDD, BTAD, SDD, and DAGM.
  • Medical Datasets: BrainMRI, LiverCT, RESC, Br35H, COVID-19, and HeadCT.

The paper notes that for the Br35H, COVID-19, and HeadCT datasets, pixel-level anomaly segmentation ground truth was not available, so anomaly classification was performed instead.

Key Insights and Ideas

  • The power of LLMs and VQA for prompt generation: The core insight is that by combining the broad knowledge of LLMs with the specific visual understanding of VQA systems, we can create a rich knowledge base that significantly improves the quality and relevance of learned anomaly prompts. This moves beyond simple text templates to a more dynamic and informed prompt generation process.
  • Bridging the global-local gap in CLIP: CLIP excels at global image understanding but struggles with fine-grained local details. The paper effectively addresses this by introducing V-V attention for local feature refinement and Bi-CMCI for multi-level cross-modal interaction, demonstrating that integrating local visual semantics is key to superior anomaly segmentation.
  • Learnable prompts are superior to fixed prompts: The results strongly suggest that learnable prompts, guided by knowledge, offer much better generalization than fixed, manually crafted prompts, especially for unseen anomaly classes. This is a crucial step towards truly generalizable ZSAD.
  • Joint optimization is key: The combination of knowledge-driven prompt learning, enhanced visual feature extraction, and sophisticated cross-modal fusion, all optimized together, leads to state-of-the-art performance.

Key Results

KAnoCLIP achieves state-of-the-art performance across all 12 industrial and medical anomaly detection datasets.

  • On the MVTec-AD dataset, KAnoCLIP improved pixel-level AUC by 3.2% and image-level AUC by 1.3% compared to previous methods.
  • On the VisA dataset, pixel-level AUC improved by 2.2% and image-level AUC by 1.5%.
  • In medical datasets, significant improvements were observed, particularly in BrainMRI (image-level AUC increase of 2.1%) and LiverCT (image-level AUC increase of 4.0%, pixel-level AUC increase of 2.4%).
  • The ablation study confirms that each module (KnPL, CLIP-VV, Bi-CMCI, Conv-Adapter) contributes to the overall performance, with KnPL providing the most substantial improvement.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the use of LLMs and VQA systems for prompt generation and the multi-stage cross-attention mechanisms likely increase computational requirements compared to simpler ZSAD methods.
  • Reliance on Pre-trained Models: The framework builds upon pre-trained CLIP models. While this is a common practice, its performance is inherently tied to the quality and biases of these foundational models.
  • Future Work: The paper suggests that KAnoCLIP's success in industrial and medical domains opens avenues for exploring its application in other areas requiring robust zero-shot anomaly detection. Further research could focus on optimizing the prompt generation process for even greater efficiency and exploring more advanced cross-modal interaction mechanisms.

Summary updated

BibTeX
@inproceedings{li2025kanoclip,
  author = {Chengyuan Li and Suyang Zhou and Jieping Kong and Lei Qi and Hui Xue},
  title = {{KAnoCLIP: Zero-Shot Anomaly Detection through Knowledge-Driven Prompt Learning and Enhanced Cross-Modal Integration}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888834},
}
Figure