GPT-4V-AD: Exploring Grounding Potential of VQA-Oriented GPT-4V for Zero-Shot Anomaly Detection

Jiangning Zhang, Haoyang He, Xuhai Chen, Zhucun Xue, Yabiao Wang, Chengjie Wang, Lei Xie, Yongxin LiuCommunications 2024 · DOI · OpenAlex

Introduction

In industrial manufacturing, the detection of defects—or anomaly detection (AD)—is a critical component of quality control. Traditional deep learning approaches often require extensive datasets of labeled "anomalous" examples, which are difficult and expensive to acquire. This challenge has catalyzed research into Zero-shot Anomaly Detection, where models are expected to identify defects without prior exposure to specific failure cases.

Large Multimodal Models (LMMs), such as GPT-4V, have demonstrated remarkable semantic intelligence across various domains. However, applying them to industrial inspection reveals a significant "grounding" problem: while these models can identify that an object is defective (the "what"), they struggle to pinpoint the exact spatial coordinates of the defect (the "where"). This paper evaluates the GPT-4V-AD framework, a novel approach designed to bridge the gap between GPT-4V’s high-level reasoning and the pixel-level precision required for industrial AD.

Methodology: The GPT-4V-AD Framework

The authors propose a framework that treats anomaly detection as a Visual Question Answering (VQA) task. To overcome GPT-4V's inherent spatial limitations, the process is divided into three distinct stages:

1. Granular Region Division

Since GPT-4V cannot reliably output precise pixel coordinates, the framework pre-processes images into labeled regions. The researchers compared three methods for this division:

  • Uniform Grids: Simple but ignores object geometry.
  • Semantic Segments (SAM): Uses the Segment Anything Model, which often over-segments or misses subtle textures.
  • Super-pixels: This method proved most effective, as super-pixels respect the natural boundaries and textures of the object, providing a stable foundation for the model to analyze.

2. Structured Prompting

The model is presented with an image where each super-pixel region is outlined and assigned a unique number. The prompt instructs GPT-4V to adopt the persona of an industrial inspector. The model is tasked with evaluating each numbered region and assigning an anomaly score ranging from 0 to 1.

3. Text2Segmentation

The final stage involves parsing the textual output from GPT-4V (e.g., "region 18: 0.87"). These scores are then mapped back onto the original image coordinates to generate a visual heatmap, effectively translating the model's semantic judgment into a localized anomaly map.

Experimental Evaluation

The framework was evaluated on two primary benchmark datasets:

  • MVTec AD: The industry standard, featuring 15 categories ranging from textures (carpet, leather) to structured objects (screws, transistors).
  • VisA: A more complex dataset involving challenging structures and multiple object instances, such as printed circuit boards (PCBs) and grouped items.

Key Results

Dataset Image-level AU-ROC Pixel-level AU-ROC
VisA 88.0 76.6
MVTec AD 77.1 68.0

On the VisA dataset, GPT-4V-AD achieved a breakthrough, surpassing the previous state-of-the-art (CLIP-AD) by 6.8 points. However, performance on MVTec AD (77.1 AU-ROC) trailed behind specialized models like WinCLIP (91.8 AU-ROC). This suggests that while GPT-4V excels at complex structural reasoning, it still struggles with the extremely fine-grained texture variations typical of the MVTec dataset.

Discussion and Insights

The Semantic vs. Spatial Gap

The study confirms that GPT-4V’s strength lies in its semantic reasoning. By outsourcing spatial reasoning to a pre-processing "map" (the super-pixels), the framework allows the LMM to focus on identifying anomalies within defined boundaries. Super-pixels emerged as the "sweet spot" for this task, capturing structural layouts more consistently than SAM.

The Reference Image Paradox

One of the most surprising findings was that providing GPT-4V with "normal" reference images—a common few-shot learning technique—actually degraded performance. The additional information appeared to "disturb" the model's reasoning process, leading to less accurate responses. This suggests that GPT-4V’s zero-shot internal knowledge is currently more robust for AD than its ability to perform in-context learning for this specific application.

Limitations and Future Directions

Despite its potential, several hurdles remain for the deployment of GPT-4V in industrial settings:

  • Inference Constraints: The cost and latency associated with GPT-4V make it currently impractical for high-speed production lines compared to lightweight, specialized models.
  • Stochasticity: The inherent randomness of LMMs can lead to inconsistent anomaly scores across multiple runs, posing a challenge for industrial reliability.
  • Small Defect Detection: The labeling scheme (placing numbers within regions) can occasionally obscure very small defects, leading to false negatives.

Future research may focus on hybrid approaches that combine the semantic power of LMMs with the efficiency of CLIP-based models, or fine-tuning LMMs specifically on industrial datasets to improve their sensitivity to fine-grained anomalies.

Conclusion

This research provides the first comprehensive look at GPT-4V’s capabilities in zero-shot anomaly detection. While not yet a replacement for specialized industrial models, GPT-4V-AD demonstrates that with the right structural prompting and region-based grounding, LMMs can achieve state-of-the-art results on complex, multi-object datasets.

Summary updated

BibTeX
@inproceedings{zhang2024gpt,
  author = {Jiangning Zhang and Haoyang He and Xuhai Chen and Zhucun Xue and Yabiao Wang and Chengjie Wang and Lei Xie and Yongxin Liu},
  title = {{GPT-4V-AD: Exploring Grounding Potential of VQA-Oriented GPT-4V for Zero-Shot Anomaly Detection}},
  booktitle = {Communications},
  year = {2024},
  doi = {10.1007/978-981-97-9003-6\_1},
}

Loading related content...

Figure