Model Selection of Anomaly Detectors in the Absence of Labeled Validation Data

Clement Fung, Chen Qiu, Aodong Li, Maja Rudolph

Here is a concise, engaging summary of the paper, tailored for researchers and engineers interested in anomaly detection and foundation models.

TL;DR

Deploying zero-shot anomaly detection models is often a gamble because real-world anomalies are too rare to form a reliable validation set. To solve this, the authors introduce SWSA (Selection With Synthetic Anomalies), a framework that generates artificial anomalies from a tiny handful of normal images using diffusion models or data augmentation. SWSA enables accurate model and prompt selection without requiring a single real-world labeled anomaly.

What Problem Does This Paper Address?

Anomaly detection (AD) is critical for everything from industrial quality control to medical diagnosis. While modern foundation models (like CLIP or large ViTs) show incredible zero-shot performance, picking the right model architecture, hyperparameters, or text prompts for a specific task remains a massive challenge.

Standard machine learning relies on a labeled validation set to make these choices. However, in anomaly detection, labeled validation data is rarely available because anomalies are, by definition, rare and unpredictable. Previous work tried using completely random external datasets (like Tiny-ImageNet) as proxy anomalies, but these are too easily distinguished from normal data to be useful for fine-grained model selection.

Methods

The authors propose SWSA, a pipeline designed to build a synthetic validation dataset using only a small support set of normal images ($X_{support}$), which can be as small as 10 samples.

Overview of the SWSA framework for model selection using synthetic anomalies.

The framework operates in four steps:

  1. Partitioning: The normal support set is split into seed images and normal validation images.
  2. Outlier Generation: The seed images are used to generate synthetic anomalies via two distinct strategies:
  3. CutPaste: A simple data augmentation technique where a random region of an image is cropped and pasted onto another region of the same image.
  4. Diffusion-based Generation (DiffStyle): A novel application of a pre-trained, non-fine-tuned ImageNet diffusion model. By interpolating between the latent spaces ($h$-space) of two normal images, it generates images that retain realistic backgrounds and textures but introduce semantic corruptions.
  5. Mixing: The synthetic anomalies are combined with the held-out normal validation images to create a labeled validation set.
  6. Evaluation: Candidate models (or CLIP prompts) are evaluated on this synthetic dataset using AUROC to select the optimal configuration.
Examples of synthetic anomalies generated via latent space interpolation using a pre-trained diffusion model.

Datasets

The benchmark evaluates SWSA across 329 distinct anomaly detection tasks spanning four widely used image datasets:

  • Natural Images: Caltech-UCSD Birds (CUB) and Oxford Flowers.
  • Industrial Defects: MVTec Anomaly Detection (MVTec-AD) and Visual Anomaly Detection (VisA).

Tasks include both standard "one-vs-rest" setups and the more challenging "one-vs-closest" setups (near-anomaly detection).

Key Insights and Ideas

  • No Domain-Specific Fine-Tuning Needed: Remarkably, a general-purpose diffusion model trained purely on ImageNet can generate highly effective synthetic anomalies for entirely different domains (like specific bird or flower species) without any fine-tuning.
  • The "Right" Anomaly Depends on the Domain: The optimal generation strategy depends heavily on the nature of the dataset.
  • For natural images (CUB, Flowers), anomalies represent semantic shifts (e.g., a different bird species). Here, diffusion-based interpolation shines because it creates plausible new semantic variations.
  • For industrial images (MVTec, VisA), anomalies are subtle, local structural defects (e.g., a broken wire). For these tasks, the localized pixel-level corruptions of CutPaste provide a much better proxy.
t-SNE visualization showing how diffusion-generated anomalies align well with natural variations, while CutPaste aligns better with local industrial defects.

Key Results

  • Accurate Model Selection: When tasked with selecting the best model out of 10 candidate architectures (ResNets and ViTs), SWSA successfully selected the true best-performing model in 7 out of 8 experimental settings.
  • Outperforming Baselines: SWSA consistently beat baseline strategies, such as blindly choosing the largest available model (ViT-H-14) or using Tiny-ImageNet as an outlier proxy.
  • CLIP Prompt Selection: For zero-shot CLIP deployment, SWSA successfully navigated a pool of candidate prompt templates, outperforming both the default prompt and standard prompt ensembling techniques in natural image settings.

Limitations and Future Work

While SWSA is highly effective for natural images, its performance drops on complex industrial datasets (MVTec-AD and VisA). As shown in the t-SNE visualization, real industrial anomalies cluster incredibly close to normal data, making them exceptionally difficult to mimic artificially.

Additionally, the authors note that calculating the total variation distance between synthetic and real validation sets does not yield a tight theoretical bound for rank preservation. Bridging this gap between strong empirical success and theoretical guarantees remains an open avenue for future research.

Summary updated

Abstract

Anomaly detection requires detecting abnormal samples in large unlabeled datasets. While progress in deep learning and the advent of foundation models has produced powerful unsupervised anomaly detection methods, their deployment in practice is often hindered by the lack of labeled data---without it, the detection accuracy of an anomaly detector cannot be evaluated reliably. In this work, we propose a general-purpose framework for evaluating image-based anomaly detectors with synthetically generated validation data. Our method assumes access to a small support set of normal images which are processed with a pre-trained diffusion model (our proposed method requires no training or fine-tuning) to produce synthetic anomalies. When mixed with normal samples from the support set, the synthetic anomalies create detection tasks that compose a validation framework for anomaly detection evaluation and model selection. In an extensive empirical study, ranging from natural images to industrial applications, we find that our synthetic validation framework selects the same models and hyper-parameters as selection with a ground-truth validation set. In addition, we find that prompts selected by our method for CLIP-based anomaly detection outperforms all other prompt selection strategies, and leads to the overall best detection accuracy, even on the challenging MVTec-AD dataset.
BibTeX
@article{fung2025model,
  author = {Clement Fung and Chen Qiu and Aodong Li and Maja Rudolph},
  title = {{Model Selection of Anomaly Detectors in the Absence of Labeled Validation Data}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2025},
  doi = {10.1109/tai.2025.3562505},
}
Figure