Zero-Shot Semantic Anomaly Detection

25 concepts19 papers2 proposals

Traditional anomaly detection (AD) treats outliers as statistical deviations—points that fall into low-density regions of a feature space. However, recent research suggests a fundamental shift: identifying anomalies by reasoning about their semantic meaning rather than just their statistical distribution. By leveraging foundation models like CLIP and GPT-4, researchers are now able to detect novel defects and "out-of-distribution" (OOD) events without any task-specific training.

From Statistical Distance to Semantic Alignment

In classical unsupervised AD, a model learns the "normal" distribution and flags anything far away from it. The emerging paradigm of zero-shot semantic anomaly detection replaces this distance-based logic with alignment-based logic.

As Philipp Liznerski et al. demonstrate in Exposing Outlier Exposure, foundation models pre-trained on massive multimodal datasets already possess an internal "map" of what constitutes a defect. Instead of training a model on thousands of "normal" images, we can simply ask a vision-language model (VLM) how well an image aligns with a text prompt like "a photo of a damaged component" versus "a photo of a perfect component."

This works because models like CLIP have seen millions of examples of "scratched," "broken," or "missing" objects during their general pre-training. Jongheon Jeong et al. formalize this in WinCLIP, using "compositional prompt ensembles"—multiple descriptive strings that describe various states of an object—to activate the model's latent knowledge of physical defects.

Anchoring with Textual Priors

A key challenge in OOD detection is distinguishing between "near-OOD" samples (e.g., a different breed of dog) and "far-OOD" samples (e.g., a picture of a car). Stanislav Fort et al. found that large-scale transformers are surprisingly effective at this when they use textual anchors. By comparing an input embedding to the semantic representations of both in-distribution and potential outlier class names, the model uses linguistic priors to draw sharper boundaries than visual features alone could provide.

This concept is extended to specialized domains by incorporating expert knowledge into the prompts. For instance, Abd El Rahman Shabayek et al. achieved zero-shot detection in battery thermal images by providing models with prompts containing prior knowledge of thermal physics. Similarly, Jeonghyeon Kim and Sangheum Hwang propose using auxiliary negative labels—mining a broad set of "not-target" concepts—to explicitly identify when an input aligns more closely with an unknown category than a known one.

Reasoning Over Context and Time

Semantic anomalies are often "contextual": an object might look perfectly normal in isolation but be anomalous due to its environment. Amine Elhafsi et al. illustrate this with the example of a stop sign printed on a billboard. A traditional detector sees a "stop sign" and flags it as normal; a Large Language Model (LLM), however, can reason that a stop sign appearing on a billboard above a highway is a semantic inconsistency.

This reasoning capability allows for a transition from binary anomaly scores to descriptive explanations:

  • Video Anomaly Detection: Luca Zanella et al. (LAVAD) and Jiaqi Tang et al. (HAWK) treat video AD as a linguistic reasoning task. They use VLMs to caption video frames and then use LLMs to identify temporal anomalies (e.g., "a person is running in a library") through natural language reasoning.
  • Media Forensics: Shan Jia et al. show that LLMs can detect DeepFakes by identifying semantic inconsistencies, such as anatomical errors or lighting mismatches, rather than just looking for pixel-level artifacts.
  • Tabular Data: In ReTabAD, Sanghyu Yoon et al. demonstrate that providing LLMs with the semantic context of table headers (e.g., "Heart Rate" vs. "Age") allows them to identify implausible feature combinations that purely numerical models might miss.

The Shift to Open-Vocabulary Systems

The ultimate goal of this research direction is open-vocabulary detection. As surveyed by Yang Wang et al. and Arturo Mendoza et al., the field is moving away from "closed" systems that only know a few types of errors. By mapping visual features into a shared semantic space with text, systems can now identify and describe novel security incidents or industrial defects using natural language prompts, effectively leveraging the "world knowledge" of foundation models to handle the infinite variety of the real world.

Key Foundational Work

Exposing Outlier Exposure: What Can Be Learned From Few, One, and Zero Outlier Images (2022) — contributed 2 concept elements to this direction.
Exploring the Limits of Out-of-Distribution Detection (2021) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_zero_shot_semantic_anomaly_detection_22ed5ac6,
  title = {zero-shot semantic anomaly detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-semantic-anomaly-detection/rdr_22ed5ac6e55e439b8fefb272d3e129be},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_zero_shot_semantic_anomaly_detection_22ed5ac6_v1,
  title = {zero-shot semantic anomaly detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-semantic-anomaly-detection/rdr_22ed5ac6e55e439b8fefb272d3e129be/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake