A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision

Stefano Samele, Eugenio Lomurno, Teodora Jovanovic, Sanjay Shivakumar Manohar, Alberto Crivellaro, Matteo MatteucciarXiv 2026 · arXiv

TL;DR

While modern vision-language models (VLMs) claim to offer "text-guided" industrial anomaly detection, current evaluation protocols fail to verify if these models actually follow textual instructions. This paper introduces TGAD (Text-Guided Anomaly Detection), a structured benchmark revealing that current state-of-the-art models treat text merely as a static visual prior rather than a controllable command, causing their decision-making capabilities to collapse under realistic industrial constraints.

What Problem Does This Paper Address?

Industrial anomaly detection (AD)—identifying defects like scratches, missing components, or structural flaws—has traditionally been a purely visual task. Recently, the rise of Vision-Language Models (VLMs) like CLIP has birthed a new class of "text-guided" or "instruction-driven" AD tools. These models promise zero-shot or few-shot capabilities, supposedly allowing an engineer to control inspection via simple natural language prompts (e.g., "Inspect only the blue cable").

However, the authors point out a glaring flaw in how these models are evaluated: standard benchmarks (like MVTec AD) keep the text prompt constant across test samples. This means we don't actually know if a model's high accuracy stems from it dynamically following text instructions, or if it is simply relying on powerful, pretrained visual features. This paper investigates whether language actually conditions the model's final decision.

Methods

The authors introduce the TGAD benchmark, which evaluates models across three scenarios that progressively increase the functional role of language:

  1. Scenario 1: Controlled Text-Sensitivity: Keeps the image fixed but systematically alters the prompt (e.g., removing descriptions, removing normality assertions, or stripping it down to a minimal query) to see if the model's anomaly map actually changes.
  2. Scenario 2: Per-Part Anomaly Detection: Instructs the model to inspect only a specific component of an object. Crucially, under a strict evaluation setting (EV2), images containing defects outside the instructed component are treated as normal. This tests whether the model can selectively ignore irrelevant visual anomalies based on text.
  3. Scenario 3: Industrial Robustness: Combines both component-location and defect-type constraints in a highly complex, cluttered industrial environment.

The benchmark evaluates three distinct VLM paradigms: generative large vision-language (AnomalyGPT), training-free discriminative (LogSAD), and embedding-adaptive discriminative (AA-CLIP). To enable per-part inspection for LogSAD, the authors also introduce a novel foreground-masking extension (FMS).

Figure 1: Overview of the three TGAD scenarios, ordered by the functional role language must play in the decision. In each, a fixed image and a textual request are processed by a multimodal anomaly detection model and scored against the ground truth. Scenario 1 varies the prompt on MVTec AD to test whether the output moves with language; Scenario 2 instructs inspection of a single component, admitting defects in other components as normal; Scenario 3 uses the Assembled Panel dataset, where both the defect type and the component location must be known to inspect.

Datasets

The benchmark utilizes three datasets:

  • MVTec AD: Used as-is for Scenario 1 to establish baseline prompt sensitivity.
  • Component-Tagged MVTec AD: A new extension introduced by the authors that adds component-level region tags (e.g., light-blue insulation, green cable, gray cable) to every defective test image across 9 object classes.
  • Assembled Panel Dataset (APD): A brand-new, highly challenging dataset developed with an industrial partner. It consists of 99 defect-free and 53 pixel-annotated anomalous images of complex electronic panels for electric-vehicle charging stations, featuring 14 distinct anomaly categories (e.g., swapped cable colors, missing screws, unplugged wires).

Key Insights and Ideas

  • Object-Anchor Collapse: VLMs are overwhelmingly dependent on the object noun itself. In Scenario 1, changing descriptive details or removing lists of potential flaws had virtually zero impact on performance. The models only reacted when the object noun was entirely removed, proving they treat text as a static label rather than an instruction.
  • Localization-Decision Dissociation: When models are told to focus on a specific sub-component, they can often still visually localize a defect (maintaining a high localization score like AUPRO). However, they fail to update their image-level classification decision—meaning they still flag the image as anomalous even if the defect is in a region they were explicitly told to ignore.
  • Text as a Prior, Not a Control Signal: Ultimately, current multimodal anomaly detectors use language merely as a weak class prior. They can find what they already know how to find, but they cannot use text as a dynamic control signal to alter their final decision.

Key Results

  • Scenario 1: AnomalyGPT’s image-level AUROC (I-AUROC) remained completely flat at 97.4 across four distinct prompt variations, only dropping to 82.6 when the object noun was deleted.
  • Scenario 2: When forced to ignore irrelevant defects (EV2), performance plummeted. AnomalyGPT’s one-part I-AUROC dropped from 90.3 to 66.3. LogSAD dropped from 97.6 to 73.0.
  • Scenario 3 (APD): On the complex Assembled Panel Dataset, image-level discrimination collapsed entirely. Under per-error prompting, LogSAD achieved an I-AUROC of just 50.5 (pure chance), while AA-CLIP collapsed to 31.5 (worse than chance), despite both maintaining reasonable pixel-level localization.
Per-model performance degradation across the three TGAD scenarios

Limitations and Future Work

The primary limitation highlighted is architectural: current VLM pipelines pass text into early feature extraction or localization layers, but the final image-level decision mechanism never actually "sees" or conditions on the text constraint.

The authors suggest that future work must move away from simple object-naming paradigms and toward scoring architectures that explicitly bind the final classification decision to linguistic constraints. Furthermore, the EV2 evaluation setting introduced in this paper can serve as a training curriculum to teach future models true instruction-following capabilities.

Summary updated

Abstract

Industrial anomaly detection has historically been a unimodal task. Recent multimodal vision-language models have produced systems that admit textual input alongside the image and are presented as enabling text-guided zero- and few-shot inspection. Yet these methods are evaluated with protocols inherited from unimodal benchmarks that hold the textual condition constant and therefore cannot measure whether language conditions the decision; whether reported gains reflect text guidance or strong pretrained visual features remains open. We introduce Text-Guided Anomaly Detection (TGAD), a structured benchmark that progressively increases the functional role of language across three scenarios: a controlled prompt-sensitivity setting on MVTec AD; a component-tagged extension of MVTec AD that requires the model to restrict its assessment to an instructed part; and the new Assembled Panel Dataset (APD), a realistic industrial setting that requires both defect-type and component-location knowledge. We evaluate one representative model per paradigm: generative large vision-language, training-free discriminative, and embedding-adaptive discriminative. In all three, the textual interface conditions the decision only superficially: prompt content is absorbed unless the object noun is removed (the generative model's I-AUROC drops from 97.4 to 82.6); component-level instructions do not constrain the decision once defects outside the instructed part are admitted as normal (from 90.3 to 66.3); and when both combine on APD, image-level discrimination collapses below the MVTec level, in one case below chance (71.2, 50.5, 31.5). These results suggest that standard benchmarks overstate the text-guided capabilities of current multimodal anomaly detection systems, and that a protocol of this kind is a prerequisite for models that can be reliably controlled through language for industrial deployment.
BibTeX
@misc{samele2026structured,
  author = {Stefano Samele and Eugenio Lomurno and Teodora Jovanovic and Sanjay Shivakumar Manohar and Alberto Crivellaro and Matteo Matteucci},
  title = {{A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision}},
  year = {2026},
  eprint = {2606.01992},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.01992},
}
Figure