Abstract
Anomaly detection requires detecting abnormal samples in large unlabeled datasets. While progress in deep learning and the advent of foundation models has produced powerful unsupervised anomaly detection methods, their deployment in practice is often hindered by the lack of labeled data -- without it, the detection accuracy of an anomaly detector cannot be evaluated reliably. In this work, we propose a general-purpose framework for evaluating image-based anomaly detectors with synthetically generated validation data. Our method assumes access to a small support set of normal images which are processed with a pre-trained diffusion model (our proposed method requires no training or fine-tuning) to produce synthetic anomalies. When mixed with normal samples from the support set, the synthetic anomalies create detection tasks that compose a validation framework for anomaly detection evaluation and model selection. In an extensive empirical study, ranging from natural images to industrial applications, we find that our synthetic validation framework selects the same models and hyper-parameters as selection with a ground-truth validation set. In addition, we find that prompts selected by our method for CLIP-based anomaly detection outperforms all other prompt selection strategies, and leads to the overall best detection accuracy, even on the challenging MVTec-AD dataset.
Introduction
Anomaly detection (AD) is critical for identifying rare, unexpected events such as manufacturing defects or fraudulent transactions. However, AD presents a unique challenge: because anomalies are rare by definition, practitioners often lack labeled validation data to evaluate and select the best model for a specific task. Without a reliable yardstick, model selection often relies on heuristics or the assumption that larger models will perform better, which is not always the case. This work introduces SWSA (Selection With Synthetic Anomalies), a framework designed to solve the model selection problem using only examples of "normal" data.
Methodology: Selection With Synthetic Anomalies (SWSA)
The SWSA framework generates a synthetic validation set to predict how an anomaly detector will perform on real-world data. The process involves four key stages:
- Partitioning: A small support set of normal images (as few as 10–20) is split into "seeds" and "validation normals."
- Generation: Synthetic anomalies are created from the seed images. The framework utilizes two primary techniques:
- CutPaste: A classic augmentation where image patches are cropped and pasted elsewhere, effectively mimicking local, physical defects.
- Diffusion-based (DiffStyle): A novel approach using a pre-trained diffusion model (e.g., trained on ImageNet) to interpolate between normal images in the latent space. This creates realistic images with semantic corruptions without requiring any fine-tuning of the diffusion model.
- Mixing: These synthetic anomalies are combined with the held-out normal images to form a labeled validation set.
- Evaluation and Selection: Candidate models are evaluated on this synthetic set. The model achieving the highest performance is selected for deployment.
Experimental Evaluation
The effectiveness of SWSA was evaluated across 329 anomaly detection tasks using both natural image datasets (CUB, Oxford Flowers) and industrial datasets (MVTec-AD, VisA).
A key finding is the strong correlation between performance on synthetic validation data and performance on real-world ground-truth anomalies. As shown in the figure below, the synthetic AUROC serves as a reliable proxy for real validation AUROC across various datasets.
Key results include:
- High Selection Accuracy: SWSA correctly identified the best-performing model in 7 out of 8 experimental settings.
- Superiority over Baselines: The framework consistently outperformed standard strategies like "outlier exposure," which uses generic datasets (e.g., Tiny-ImageNet) as proxies for anomalies.
- Zero-Shot Prompt Selection: When applied to CLIP, SWSA successfully selected optimal text prompts for anomaly detection without ever seeing a real defect, outperforming all other prompt selection strategies.
Discussion and Insights
The research highlights that the choice of synthetic anomaly generation should be domain-specific. Diffusion-based anomalies are highly effective for natural images where anomalies are semantic (e.g., a different bird species). Conversely, the simpler CutPaste method often performs better for industrial tasks where anomalies are typically local physical defects like scratches or cracks.
Furthermore, the study demonstrates that "bigger isn't always better." SWSA was able to identify smaller, specialized models that outperformed massive vision transformers on specific tasks, proving that data-driven model selection is vital even when using powerful foundation models.
Conclusion
SWSA provides a robust, general-purpose framework for anomaly detection evaluation in the absence of labeled data. By leveraging pre-trained diffusion models and simple augmentations, it enables practitioners to make informed model selection decisions, bridging the gap between unsupervised training and reliable deployment. Future work will focus on refining anomaly generation for complex industrial defects and tightening the theoretical bounds between synthetic and real-world performance.
Summary updated