Traditional visual anomaly detection operates under a strict paradigm: a model must be trained on a large, curated dataset of "normal" images for a specific object category so it can learn to flag deviations. While effective in controlled environments, this approach breaks down in real-world deployment. As noted by Wenbing Zhu et al. in their documentation of industrial datasets, collecting category-specific training data for every new product line is a massive operational bottleneck. Furthermore, as Shengze Li et al. point out, the visual characteristics of defects are inherently unpredictable and infinite; you cannot train a model on every potential failure mode.
To bypass this data dependency, a new research direction has emerged: Zero-Shot Anomaly Detection. The goal is to build generalist models capable of identifying anomalies or out-of-distribution objects in entirely novel categories immediately upon deployment, without any category-specific training or fine-tuning.
This paradigm works by shifting the problem from learning a specific distribution to exploiting generalized visual priors and dynamic self-referencing.
Exploiting Foundation Model Priors
Why can a model detect a defect in an object it has never seen before? The answer lies in the rich, pre-trained representations of vision-language and segmentation foundation models. These models have already mapped the boundaries of "objectness" and the semantic relationships of the visual world.
Semantic Alignment via Language
Large-scale vision-language models like CLIP have learned a shared embedding space for text and images. As highlighted in surveys by Tianyi Zhao et al. and Jing Ren et al., this allows users to define the boundaries of normal and anomalous behavior using natural language prompts.
For example, Shengze Li et al. developed a framework that uses CLIP to semantically locate anomalies via text prompts and then feeds those regions as "prompts" into the Segment Anything Model (SAM) to delineate precise defect boundaries. Similarly, Jeonghyo Song et al. leverage the chain-of-thought reasoning capabilities of large language models to analyze dense scenes and identify out-of-distribution objects based on logical context.
Repurposing Geometric and Mask Priors
Foundation models designed for segmentation also contain latent anomaly-detection capabilities. Because these models are trained to recognize discrete object boundaries across millions of diverse images, their internal representations naturally flag irregularities.
[Input Image] ---> [Segmentation Model] ---> [Raw Mask Outputs] ---> [Uncertainty Map]
|
v
[Discrete Anomaly Mask]
The authors of Maskomaly demonstrate that standard mask-based segmentation networks retain raw mask outputs that are typically discarded during inference. By analyzing these raw outputs, they can extract sufficient information to distinguish between known and unknown classes without explicit exposure to anomalies.
Taking this a step further, Alexey Nekrasov 0001 et al. use pixel-level uncertainty maps from a primary segmentation model to generate spatial prompts for SAM. This converts vague anomaly "heatmaps" into distinct, isolated object instances, allowing autonomous systems to identify unexpected obstacles on the fly.
Defining Normality On the Fly
When language prompts or explicit category labels are unavailable, zero-shot models must deduce what is "normal" purely from the context of the test data. Researchers have developed three primary mechanisms to achieve this: in-context learning, single-image prototyping, and batch distribution modeling.
In-Context Residual Learning
Instead of freezing a model's understanding of normality during training, Guansong Pang et al. propose using a few normal images as dynamic, "in-context" prompts. The model evaluates a target image by calculating its deviation—the "residual"—from these provided exemplars. This allows the system to adapt its definition of normality to entirely new domains (e.g., switching from industrial parts to medical scans) in seconds, without updating any network weights.
Intrinsic Normal Prototypes
What if you don't even have a few reference images? Wei Luo et al. show that the reference can be found within the test image itself. Their framework, INP-Former, extracts "Intrinsic Normal Prototypes" directly from the uncorrupted regions of a single target image. Because these prototypes share the exact same lighting, pose, and background as the rest of the image, comparing the image against itself provides a highly accurate, perfectly aligned baseline for defect localization.
Unlabeled Batch Distribution Modeling
In industrial settings, items are often processed in batches. Xurui Li et al. exploit a fundamental statistical reality of manufacturing: in any given batch of unlabeled images, normal features are highly redundant and common, whereas anomalies are rare and unique.
Unlabeled Batch:
[ Normal Item ] --> High Feature Redundancy \ Mutual Scoring
[ Normal Item ] --> High Feature Redundancy --> Establishes "Normality"
[ Defective ] --> Rare, Unique Feature / Flagged as Anomaly
By implementing a mutual scoring mechanism across the unlabeled batch, their framework treats the batch itself as an implicit distribution of normality. This completely eliminates the need for manually engineered text descriptions or pre-curated training sets.
Why It Works: A Shift in Perspective
Zero-shot anomaly detection succeeds because it replaces the rigid, narrow boundaries of traditional one-class classifiers with flexible, relational reasoning.
- Category Agnosticism: Models like the one proposed by Chaoqin Huang et al. focus on learning how to compare images (e.g., via image registration proxy tasks) rather than memorizing what specific objects look like.
- Logical vs. Structural Analysis: As shown by Yun Peng et al., treating a scene as a collection of discrete objects rather than a grid of pixels allows models to catch "logical anomalies"—such as a missing component or an incorrect arrangement—even when the individual components look perfectly normal.
By combining the broad, open-vocabulary understanding of foundation models with localized, self-referencing statistics, zero-shot architectures transform anomaly detection from a data-heavy training problem into an algorithmic inference problem.
Key Foundational Work
Multimodal Detection of Unknown Objects on Roads for Autonomous Driving (2022) — contributed 2 concept elements to this direction.
DeepGD3: Unknown-Aware Deep Generative/Discriminative Hybrid Defect Detector for PCB Soldering Inspection (2023) — contributed 1 concept elements to this direction.
UGainS: Uncertainty Guided Anomaly Instance Segmentation (2023) — contributed 1 concept elements to this direction.