TL;DR
While modern vision-language models (VLMs) claim to offer "text-guided" industrial anomaly detection, current evaluation protocols fail to verify if these models actually follow textual instructions. This paper introduces TGAD (Text-Guided Anomaly Detection), a structured benchmark revealing that current state-of-the-art models treat text merely as a static visual prior rather than a controllable command, causing their decision-making capabilities to collapse under realistic industrial constraints.
What Problem Does This Paper Address?
Industrial anomaly detection (AD)—identifying defects like scratches, missing components, or structural flaws—has traditionally been a purely visual task. Recently, the rise of Vision-Language Models (VLMs) like CLIP has birthed a new class of "text-guided" or "instruction-driven" AD tools. These models promise zero-shot or few-shot capabilities, supposedly allowing an engineer to control inspection via simple natural language prompts (e.g., "Inspect only the blue cable").
However, the authors point out a glaring flaw in how these models are evaluated: standard benchmarks (like MVTec AD) keep the text prompt constant across test samples. This means we don't actually know if a model's high accuracy stems from it dynamically following text instructions, or if it is simply relying on powerful, pretrained visual features. This paper investigates whether language actually conditions the model's final decision.
Methods
The authors introduce the TGAD benchmark, which evaluates models across three scenarios that progressively increase the functional role of language:
- Scenario 1: Controlled Text-Sensitivity: Keeps the image fixed but systematically alters the prompt (e.g., removing descriptions, removing normality assertions, or stripping it down to a minimal query) to see if the model's anomaly map actually changes.
- Scenario 2: Per-Part Anomaly Detection: Instructs the model to inspect only a specific component of an object. Crucially, under a strict evaluation setting (EV2), images containing defects outside the instructed component are treated as normal. This tests whether the model can selectively ignore irrelevant visual anomalies based on text.
- Scenario 3: Industrial Robustness: Combines both component-location and defect-type constraints in a highly complex, cluttered industrial environment.
The benchmark evaluates three distinct VLM paradigms: generative large vision-language (AnomalyGPT), training-free discriminative (LogSAD), and embedding-adaptive discriminative (AA-CLIP). To enable per-part inspection for LogSAD, the authors also introduce a novel foreground-masking extension (FMS).
Datasets
The benchmark utilizes three datasets:
- MVTec AD: Used as-is for Scenario 1 to establish baseline prompt sensitivity.
- Component-Tagged MVTec AD: A new extension introduced by the authors that adds component-level region tags (e.g., light-blue insulation, green cable, gray cable) to every defective test image across 9 object classes.
- Assembled Panel Dataset (APD): A brand-new, highly challenging dataset developed with an industrial partner. It consists of 99 defect-free and 53 pixel-annotated anomalous images of complex electronic panels for electric-vehicle charging stations, featuring 14 distinct anomaly categories (e.g., swapped cable colors, missing screws, unplugged wires).
Key Insights and Ideas
- Object-Anchor Collapse: VLMs are overwhelmingly dependent on the object noun itself. In Scenario 1, changing descriptive details or removing lists of potential flaws had virtually zero impact on performance. The models only reacted when the object noun was entirely removed, proving they treat text as a static label rather than an instruction.
- Localization-Decision Dissociation: When models are told to focus on a specific sub-component, they can often still visually localize a defect (maintaining a high localization score like AUPRO). However, they fail to update their image-level classification decision—meaning they still flag the image as anomalous even if the defect is in a region they were explicitly told to ignore.
- Text as a Prior, Not a Control Signal: Ultimately, current multimodal anomaly detectors use language merely as a weak class prior. They can find what they already know how to find, but they cannot use text as a dynamic control signal to alter their final decision.
Key Results
- Scenario 1: AnomalyGPT’s image-level AUROC (I-AUROC) remained completely flat at 97.4 across four distinct prompt variations, only dropping to 82.6 when the object noun was deleted.
- Scenario 2: When forced to ignore irrelevant defects (EV2), performance plummeted. AnomalyGPT’s one-part I-AUROC dropped from 90.3 to 66.3. LogSAD dropped from 97.6 to 73.0.
- Scenario 3 (APD): On the complex Assembled Panel Dataset, image-level discrimination collapsed entirely. Under per-error prompting, LogSAD achieved an I-AUROC of just 50.5 (pure chance), while AA-CLIP collapsed to 31.5 (worse than chance), despite both maintaining reasonable pixel-level localization.
Limitations and Future Work
The primary limitation highlighted is architectural: current VLM pipelines pass text into early feature extraction or localization layers, but the final image-level decision mechanism never actually "sees" or conditions on the text constraint.
The authors suggest that future work must move away from simple object-naming paradigms and toward scoring architectures that explicitly bind the final classification decision to linguistic constraints. Furthermore, the EV2 evaluation setting introduced in this paper can serve as a training curriculum to teach future models true instruction-following capabilities.
Summary updated