CLIP-FSAC: Boosting CLIP for Few-Shot Anomaly Classification with Synthetic Anomalies

Zuo Zuo, Yao Wu, Baoqiang Li, Jiahao Dong, You Zhou, Lei Zhou, Yanyun Qu, Zongze WuIJCAI 2024 · OpenReview

Abstract

Few-shot anomaly classification (FSAC) is a vital task in the manufacturing industry. Recent methods focus on utilizing CLIP in zero/few-normal-shot anomaly detection instead of custom models. However, there is a lack of specific text prompts in anomaly classification, and most ignore the modality gap between image and text. Meanwhile, there is a distribution discrepancy between the pre-trained and the target data. To provide a remedy, we propose a method to boost CLIP for few-normal-shot anomaly classification, dubbed CLIP-FSAC, which contains two stages of training and alternating fine-tuning with two modality-specific adapters. Specifically, in the first stage, we train an image adapter with text representation output from the text encoder and introduce image-to-text tuning to enhance multi-modal interaction and facilitate a better language-compatible visual representation. In the second stage, we freeze the image adapter to train the text adapter. Both are constrained by fusion-text contrastive loss. Comprehensive experiment results show our method outperforms the state-of-the-art by 12.2%, 10.9%, and 10.4% AUROC on VisA for 1, 2, and 4-shot settings.

Introduction

Industrial quality control relies heavily on Image Anomaly Detection (IAD) to identify defects. However, collecting large-scale datasets of anomalous samples is often impractical in manufacturing. This has led to the rise of Few-Shot Anomaly Classification (FSAC), where models must learn from only a handful of normal samples. Traditional approaches often rely on meta-learning or complex memory banks to store normal features. While Vision-Language Models (VLMs) like CLIP offer a promising alternative due to their broad pre-trained knowledge, they face challenges in industrial settings: a significant modality gap between general text/images and specific industrial defects, and a distribution discrepancy between pre-training data and target industrial images.

Comparison of anomaly detection paradigms: (a) Meta Learning, (b) Memory Bank, and (c) the proposed VLM-based few-shot classification.

Methodology: The CLIP-FSAC Framework

The proposed CLIP-FSAC framework addresses these issues through a two-stage training strategy and synthetic anomaly generation. By bridging the gap between industrial images and text prompts, it allows CLIP to specialize in niche industrial vocabulary.

Synthetic Anomaly Generation

To overcome the lack of real-world defect data, CLIP-FSAC utilizes synthetic anomalies. Methods like Random Perturbation (adding Gaussian noise to random regions) and Natural Synthetic Anomalies (NSA, cloning patches from other images) are used to create "anomalous" training examples from normal ones. This teaches the model the concept of an anomaly without requiring real defect samples.

Two-Stage Training Strategy

The core of CLIP-FSAC is an alternating fine-tuning process using two modality-specific adapters. Instead of joint training, which can be unstable with limited data, the model follows a sequential approach:

  1. Stage 1 (Image Adapter): An image adapter ($A_f$) is trained while the text encoder remains frozen. An Image-to-Text Cross-Attention module is introduced, where text features act as queries to extract language-compatible visual cues from the image features.
  2. Stage 2 (Text Adapter): The image adapter is frozen, and a text adapter ($A_g$) is trained. This ensures the text prompts (e.g., "a photo of a damaged [object]") are perfectly calibrated to the newly refined visual space.
Architecture of CLIP-FSAC: (a) Zero-shot baseline (CLIP-AC) and (b) the proposed two-stage training process involving image and text adapters.

Both stages are optimized using a fusion-text contrastive loss, which pulls normal image features toward normal text prompts and pushes synthetic anomaly features toward anomalous text prompts.

Results and Discussion

CLIP-FSAC was evaluated on the MVTec-AD and VisA benchmarks in 1, 2, and 4-shot settings. On the VisA dataset, it achieved a remarkable 96.0% AUROC in the 1-shot setting, representing a 12.2% improvement over the previous state-of-the-art, WinCLIP. On MVTec-AD, it reached 95.5% AUROC, outperforming WinCLIP by 2.4%.

Key insights from the results include:

  • Sequential Tuning Matters: Stabilizing the visual representation first makes it significantly easier for the language side to align later.
  • Cross-Attention is Vital: Using text features to guide image features via cross-attention helps close the modality gap by focusing on relevant visual cues.
  • Efficiency: With only 4 normal images, CLIP-FSAC performs comparably to "full-shot" models trained on hundreds of images.

Conclusion

CLIP-FSAC demonstrates that stabilizing visual representations before aligning language prompts is a superior strategy for few-shot anomaly classification. By leveraging synthetic data and cross-modal adapters, the model achieves high performance with minimal data, making it a practical and powerful solution for modern industrial quality control.

Summary updated

Abstract

Few-shot anomaly classification (FSAC) is a vital task in manufacturing industry. Recent methods focus on utilizing CLIP in zero/few normal shot anomaly detection instead of custom models. However, there is a lack of specific text prompts in anomaly classification and most of them ignore the modality gap between image and text. Meanwhile, there is distribution discrepancy between the pre-trained and the target data. To provide a remedy, in this paper, we propose a method to boost CLIP for few-normal-shot anomaly classification, dubbed CLIP-FSAC, which contains two-stage of training and alternating fine-tuning with two modality-specific adapters. Specifically, in the first stage, we train image adapter with text representation output from text encoder and introduce an image-to-text tuning to enhance multi-modal interaction and facilitate a better language-compatible visual representation. In the second stage, we freeze the image adapter to train the text adapter. Both of them are constrained by fusion-text contrastive loss. Comprehensive experiment results are provided for evaluating our method in few-normal-shot anomaly classification, which outperforms the state-of-the-art method by 12.2%, 10.9%, 10.4% AUROC on VisA for 1, 2, and 4-shot settings.
BibTeX
@inproceedings{zuo2024clip,
  author = {Zuo Zuo and Yao Wu and Baoqiang Li and Jiahao Dong and You Zhou and Lei Zhou and Yanyun Qu and Zongze Wu},
  title = {{CLIP-FSAC: Boosting CLIP for Few-Shot Anomaly Classification with Synthetic Anomalies}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=UDDv7p7IAe},
}
Figure