Abstract
Few-shot anomaly classification (FSAC) is a vital task in the manufacturing industry. Recent methods focus on utilizing CLIP in zero/few-normal-shot anomaly detection instead of custom models. However, there is a lack of specific text prompts in anomaly classification, and most ignore the modality gap between image and text. Meanwhile, there is a distribution discrepancy between the pre-trained and the target data. To provide a remedy, we propose a method to boost CLIP for few-normal-shot anomaly classification, dubbed CLIP-FSAC, which contains two stages of training and alternating fine-tuning with two modality-specific adapters. Specifically, in the first stage, we train an image adapter with text representation output from the text encoder and introduce image-to-text tuning to enhance multi-modal interaction and facilitate a better language-compatible visual representation. In the second stage, we freeze the image adapter to train the text adapter. Both are constrained by fusion-text contrastive loss. Comprehensive experiment results show our method outperforms the state-of-the-art by 12.2%, 10.9%, and 10.4% AUROC on VisA for 1, 2, and 4-shot settings.
Introduction
Industrial quality control relies heavily on Image Anomaly Detection (IAD) to identify defects. However, collecting large-scale datasets of anomalous samples is often impractical in manufacturing. This has led to the rise of Few-Shot Anomaly Classification (FSAC), where models must learn from only a handful of normal samples. Traditional approaches often rely on meta-learning or complex memory banks to store normal features. While Vision-Language Models (VLMs) like CLIP offer a promising alternative due to their broad pre-trained knowledge, they face challenges in industrial settings: a significant modality gap between general text/images and specific industrial defects, and a distribution discrepancy between pre-training data and target industrial images.
Methodology: The CLIP-FSAC Framework
The proposed CLIP-FSAC framework addresses these issues through a two-stage training strategy and synthetic anomaly generation. By bridging the gap between industrial images and text prompts, it allows CLIP to specialize in niche industrial vocabulary.
Synthetic Anomaly Generation
To overcome the lack of real-world defect data, CLIP-FSAC utilizes synthetic anomalies. Methods like Random Perturbation (adding Gaussian noise to random regions) and Natural Synthetic Anomalies (NSA, cloning patches from other images) are used to create "anomalous" training examples from normal ones. This teaches the model the concept of an anomaly without requiring real defect samples.
Two-Stage Training Strategy
The core of CLIP-FSAC is an alternating fine-tuning process using two modality-specific adapters. Instead of joint training, which can be unstable with limited data, the model follows a sequential approach:
- Stage 1 (Image Adapter): An image adapter ($A_f$) is trained while the text encoder remains frozen. An Image-to-Text Cross-Attention module is introduced, where text features act as queries to extract language-compatible visual cues from the image features.
- Stage 2 (Text Adapter): The image adapter is frozen, and a text adapter ($A_g$) is trained. This ensures the text prompts (e.g., "a photo of a damaged [object]") are perfectly calibrated to the newly refined visual space.
Both stages are optimized using a fusion-text contrastive loss, which pulls normal image features toward normal text prompts and pushes synthetic anomaly features toward anomalous text prompts.
Results and Discussion
CLIP-FSAC was evaluated on the MVTec-AD and VisA benchmarks in 1, 2, and 4-shot settings. On the VisA dataset, it achieved a remarkable 96.0% AUROC in the 1-shot setting, representing a 12.2% improvement over the previous state-of-the-art, WinCLIP. On MVTec-AD, it reached 95.5% AUROC, outperforming WinCLIP by 2.4%.
Key insights from the results include:
- Sequential Tuning Matters: Stabilizing the visual representation first makes it significantly easier for the language side to align later.
- Cross-Attention is Vital: Using text features to guide image features via cross-attention helps close the modality gap by focusing on relevant visual cues.
- Efficiency: With only 4 normal images, CLIP-FSAC performs comparably to "full-shot" models trained on hundreds of images.
Conclusion
CLIP-FSAC demonstrates that stabilizing visual representations before aligning language prompts is a superior strategy for few-shot anomaly classification. By leveraging synthetic data and cross-modal adapters, the model achieves high performance with minimal data, making it a practical and powerful solution for modern industrial quality control.
Summary updated