Model Selection of Anomaly Detectors in the Absence of Labeled Validation Data

Clement Fung, Chen Qiu, Aodong Li, Maja Rudolph

Here is a concise, engaging summary of the paper, tailored for researchers and engineers interested in anomaly detection and foundation models.

TL;DR

Deploying zero-shot anomaly detection models is often a gamble because real-world anomalies are too rare to form a reliable validation set. To solve this, the authors introduce SWSA (Selection With Synthetic Anomalies), a framework that generates artificial anomalies from a tiny handful of normal images using diffusion models or data augmentation. SWSA enables accurate model and prompt selection without requiring a single real-world labeled anomaly.

What Problem Does This Paper Address?

Anomaly detection (AD) is critical for everything from industrial quality control to medical diagnosis. While modern foundation models (like CLIP or large ViTs) show incredible zero-shot performance, picking the right model architecture, hyperparameters, or text prompts for a specific task remains a massive challenge.

Standard machine learning relies on a labeled validation set to make these choices. However, in anomaly detection, labeled validation data is rarely available because anomalies are, by definition, rare and unpredictable. Previous work tried using completely random external datasets (like Tiny-ImageNet) as proxy anomalies, but these are too easily distinguished from normal data to be useful for fine-grained model selection.

Methods

The authors propose SWSA, a pipeline designed to build a synthetic validation dataset using only a small support set of normal images ($X_{support}$), which can be as small as 10 samples.

Figure 1: We propose two methods for generating synthetic anomalies, as described in Sec. 3.2: image-guided generation with a diffusion model and local augmentation. We produce a synthetic validation set by combining real normal images with synthetic anomalies. The synthetic validation set is then used for model selection, as described in Sec. 3.1. Components in blue are frozen, components in green are real data, and components in orange are methods implemented in this work.

The framework operates in four steps:

  1. Partitioning: The normal support set is split into seed images and normal validation images.
  2. Outlier Generation: The seed images are used to generate synthetic anomalies via two distinct strategies:
  3. CutPaste: A simple data augmentation technique where a random region of an image is cropped and pasted onto another region of the same image.
  4. Diffusion-based Generation (DiffStyle): A novel application of a pre-trained, non-fine-tuned ImageNet diffusion model. By interpolating between the latent spaces ($h$-space) of two normal images, it generates images that retain realistic backgrounds and textures but introduce semantic corruptions.
  5. Mixing: The synthetic anomalies are combined with the held-out normal validation images to create a labeled validation set.
  6. Evaluation: Candidate models (or CLIP prompts) are evaluated on this synthetic dataset using AUROC to select the optimal configuration.
Figure 2: Examples of synthetic anomalies generated with our diffusion-based method for CUB class 1 (left) and MVTec-AD “cable” (right). For each example, the top row of images (in green) are used as source “style” images, and the left column of images (in cyan) are used as source “content” images. The inner grid (in red) shows each pairwise interpolation between the source style and content image, performed with our modified DiffStyle process. All source images are drawn from the distribution of class 1 support images; no validation data or images from other classes are used.

Datasets

The benchmark evaluates SWSA across 329 distinct anomaly detection tasks spanning four widely used image datasets:

  • Natural Images: Caltech-UCSD Birds (CUB) and Oxford Flowers.
  • Industrial Defects: MVTec Anomaly Detection (MVTec-AD) and Visual Anomaly Detection (VisA).

Tasks include both standard "one-vs-rest" setups and the more challenging "one-vs-closest" setups (near-anomaly detection).

Key Insights and Ideas

  • No Domain-Specific Fine-Tuning Needed: Remarkably, a general-purpose diffusion model trained purely on ImageNet can generate highly effective synthetic anomalies for entirely different domains (like specific bird or flower species) without any fine-tuning.
  • The "Right" Anomaly Depends on the Domain: The optimal generation strategy depends heavily on the nature of the dataset.
  • For natural images (CUB, Flowers), anomalies represent semantic shifts (e.g., a different bird species). Here, diffusion-based interpolation shines because it creates plausible new semantic variations.
  • For industrial images (MVTec, VisA), anomalies are subtle, local structural defects (e.g., a broken wire). For these tasks, the localized pixel-level corruptions of CutPaste provide a much better proxy.
t-SNE visualization showing how diffusion-generated anomalies align well with natural variations, while CutPaste aligns better with local industrial defects.

Key Results

  • Accurate Model Selection: When tasked with selecting the best model out of 10 candidate architectures (ResNets and ViTs), SWSA successfully selected the true best-performing model in 7 out of 8 experimental settings.
  • Outperforming Baselines: SWSA consistently beat baseline strategies, such as blindly choosing the largest available model (ViT-H-14) or using Tiny-ImageNet as an outlier proxy.
  • CLIP Prompt Selection: For zero-shot CLIP deployment, SWSA successfully navigated a pool of candidate prompt templates, outperforming both the default prompt and standard prompt ensembling techniques in natural image settings.

Limitations and Future Work

While SWSA is highly effective for natural images, its performance drops on complex industrial datasets (MVTec-AD and VisA). As shown in the t-SNE visualization, real industrial anomalies cluster incredibly close to normal data, making them exceptionally difficult to mimic artificially.

Additionally, the authors note that calculating the total variation distance between synthetic and real validation sets does not yield a tight theoretical bound for rank preservation. Bridging this gap between strong empirical success and theoretical guarantees remains an open avenue for future research.

Summary updated

Abstract

Anomaly detection is the task of identifying abnormal samples in large unlabeled datasets. While the advent of foundation models has produced powerful zero-shot anomaly detection methods, their deployment in practice is often hindered by the absence of labeled validation data -- without it, their detection performance cannot be evaluated reliably. In this work, we propose SWSA (Selection With Synthetic Anomalies): a general-purpose framework to select image-based anomaly detectors without labeled validation data. Instead of collecting labeled validation data, we generate synthetic anomalies without any training or fine-tuning, using only a small support set of normal images. Our synthetic anomalies are used to create detection tasks that compose a validation framework for model selection. In an empirical study, we evaluate SWSA with three types of synthetic anomalies and on two selection tasks: model selection of image-based anomaly detectors and prompt selection for CLIP-based anomaly detection. SWSA often selects models and prompts that match selections made with a ground-truth validation set, outperforming baseline selection strategies.
BibTeX
@article{fung2025model,
  author = {Clement Fung and Chen Qiu and Aodong Li and Maja Rudolph},
  title = {{Model Selection of Anomaly Detectors in the Absence of Labeled Validation Data}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2025},
  doi = {10.1109/tai.2025.3562505},
}
Figure