AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization

Jingyi Liao, Yongyi Su, Rong-Cheng Tu, Zhao Jin, Wenhao Sun, Yiting Li, Dacheng Tao, Xun Xu, Xulei Yang

Abstract

While Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities across diverse domains, their application to specialized anomaly detection (AD) remains constrained by domain adaptation challenges. Existing Group Relative Policy Optimization (GRPO) based approaches suffer from two critical limitations: inadequate training data utilization when models produce uniform responses, and insufficient supervision over reasoning processes that encourage immediate binary decisions without deliberative analysis. We propose a comprehensive framework addressing these limitations through two synergistic innovations. First, we introduce a multi-stage deliberative reasoning process that guides models from region identification to focused examination, generating diverse response patterns essential for GRPO optimization while enabling structured supervision over analytical workflows. Second, we develop a fine-grained reward mechanism incorporating classification accuracy and localization supervision, transforming binary feedback into continuous signals that distinguish genuine analytical insight from spurious correctness. Comprehensive evaluation across multiple industrial datasets demonstrates substantial performance improvements in adapting general vision-language models to specialized anomaly detection. Our method achieves superior accuracy with efficient adaptation of existing annotations, effectively bridging the gap between general-purpose MLLM capabilities and the fine-grained visual discrimination required for detecting subtle manufacturing defects and structural irregularities.

Introduction

Industrial anomaly detection (AD) demands high-precision visual discrimination to identify subtle defects like hairline cracks, missing components, or structural irregularities. While general-purpose Multimodal Large Language Models (MLLMs) have shown promise in broad visual tasks, their application to specialized AD is hindered by significant domain adaptation challenges. Specifically, reinforcement learning approaches using Group Relative Policy Optimization (GRPO) often suffer from reward variance collapse in binary tasks and "spurious correctness," where models reach the right conclusion through flawed reasoning. We propose AD-FM, a framework that introduces multi-stage deliberative reasoning and fine-grained reward optimization to address these limitations.

Methodology: Multi-Stage Deliberative Reasoning

AD-FM implements a structured "Think-Rethink-Answer" workflow to guide the model through a systematic analytical process rather than immediate binary classification. This multi-stage approach forces the model to "think twice" before reaching a conclusion:

  1. Initial Spatial Analysis (<think>): The model performs a global scan to identify suspicious regions, outputting bounding box coordinates for potential anomalies.
  2. Focused Region Examination (<rethink>): The model performs a targeted analysis of the identified regions, distinguishing between genuine anomalies and benign natural variations (e.g., wood grain or lighting shadows).
  3. Integrated Decision Making (<answer>): A final classification is reached by synthesizing global context and local details.
The AD-FM framework guides the model through a multi-stage reasoning process (Think-Rethink-Answer) and utilizes a composite reward function combining localization and classification accuracy.

Fine-Grained Reward Optimization

To overcome the limitations of sparse binary rewards, we introduce a continuous reward signal that supervises both the reasoning process and the final outcome. The total reward $r_{acc}$ is a combination of classification accuracy ($r_{cls}$) and localization precision ($r_{loc}$), which utilizes Generalized Intersection over Union (GIoU) to evaluate predicted bounding boxes.

Comparison of reward landscapes: (a) The proposed fine-grained reward creates a continuous optimization surface, while (b) traditional binary rewards lead to discrete, flat landscapes that cause training stagnation.

This fine-grained approach transforms the optimization landscape from a series of flat plateaus into a rich, continuous surface. By providing feedback on spatial reasoning even when the final classification is correct, the framework significantly reduces "zero-variance" training samples—cases where the model learns nothing from a batch—by approximately 73%.

Experimental Results

We evaluated AD-FM across multiple industrial benchmarks, including MVTec AD, VisA, and GoodsAD, as well as in zero-shot cross-dataset scenarios to simulate real-world deployment.

Performance Benchmarks

AD-FM achieved an average accuracy of 73.15% across four major datasets, a 22.11% improvement over the base Qwen2.5-VL-7B model. Notably, our fine-tuned 7B model outperformed much larger proprietary models, including GPT-4.1 (71.60%) and GPT-4o-mini (64.33%), on the multi-domain benchmark.

Generalization and Robustness

In cross-dataset tests (training on MVTec and testing on unseen datasets like MPDD and DAGM), AD-FM maintained a high average performance of 84.57% without requiring threshold tuning. Qualitative analysis confirmed that the multi-stage reasoning process acts as a "safety net," allowing the model to correct initial false positives during the <rethink> stage by identifying natural textures or benign variations.

Conclusion

AD-FM demonstrates that structured reasoning and fine-grained supervision are critical for adapting MLLMs to high-precision industrial tasks. By bridging the gap between general-purpose vision-language capabilities and specialized visual discrimination, our framework provides a robust path for deploying MLLMs in real-world manufacturing and quality control environments.

Summary updated

Abstract

While Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities across diverse domains, their application to specialized anomaly detection (AD) remains constrained by domain adaptation challenges. Existing Group Relative Policy Optimization (GRPO) based approaches suffer from two critical limitations: inadequate training data utilization when models produce uniform responses, and insufficient supervision over reasoning processes that encourage immediate binary decisions without deliberative analysis. We propose a comprehensive framework addressing these limitations through two synergistic innovations. First, we introduce a multi-stage deliberative reasoning process that guides models from region identification to focused examination, generating diverse response patterns essential for GRPO optimization while enabling structured supervision over analytical workflows. Second, we develop a fine-grained reward mechanism incorporating classification accuracy and localization supervision, transforming binary feedback into continuous signals that distinguish genuine analytical insight from spurious correctness. Comprehensive evaluation across multiple industrial datasets demonstrates substantial performance improvements in adapting general vision-language models to specialized anomaly detection. Our method achieves superior accuracy with efficient adaptation of existing annotations, effectively bridging the gap between general-purpose MLLM capabilities and the fine-grained visual discrimination required for detecting subtle manufacturing defects and structural irregularities.
BibTeX
@misc{liao2025ad,
  author = {Jingyi Liao and Yongyi Su and Rong-Cheng Tu and Zhao Jin and Wenhao Sun and Yiting Li and Dacheng Tao and Xun Xu and Xulei Yang},
  title = {{AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization}},
  year = {2025},
  eprint = {2508.04175},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.04175},
}
Figure