EMIT: Enhancing MLLMs for Industrial Anomaly Detection via Difficulty-Aware GRPO

Wei Guan, Jun Lan, Jian Cao, Hao Tan, Huijia Zhu, Weiqiang WangarXiv 2025 · DOI · arXiv

Abstract

Industrial Anomaly Detection (IAD) is a critical component of manufacturing safety, yet general-purpose Multimodal Large Language Models (MLLMs) often struggle with the domain-specific nuances and data scarcity inherent in industrial settings. We present EMIT, a novel framework designed to bridge this gap. EMIT utilizes a Difficulty-Aware Group Relative Policy Optimization (GRPO) algorithm alongside heatmap-guided contrastive embeddings to enhance the reasoning and localization capabilities of MLLMs. Our approach achieves a significant 7.77% average performance improvement over baseline models across diverse industrial inspection tasks.

Introduction

Traditional IAD systems typically provide binary labels or pixel-level masks, lacking the sophisticated reasoning required for complex industrial diagnostics. While MLLMs like GPT-4V or InternVL possess advanced reasoning capabilities, their application to IAD is hindered by three primary challenges: a significant domain gap between general and industrial imagery, the scarcity of defective samples for supervised fine-tuning (SFT), and alignment failures in standard Reinforcement Learning from Human Feedback (RLHF). Specifically, algorithms like GRPO often fail on "hard" samples where the model cannot initially generate any correct responses, leaving it without a positive learning signal.

Methodology: The EMIT Framework

EMIT addresses these challenges through a combination of architectural innovations and a refined reinforcement learning strategy.

Architectural Enhancements

To handle few-shot anomaly detection, EMIT incorporates Contrastive Embeddings. By comparing a "query" image against a "reference" normal image using a shared Vision Transformer (ViT), the system generates a heatmap based on patch-level cosine distances. This heatmap is then projected into the MLLM’s embedding space, effectively highlighting anomalous regions. Additionally, learnable Soft Prompts are employed to align task-specific queries with these visual contrastive features.

The EMIT framework integrates reference/query images, domain knowledge, and contrastive embeddings into the MLLM pipeline.

Difficulty-Aware GRPO

The core of EMIT is an evolution of the GRPO algorithm. To overcome the "cold start" problem for difficult industrial samples, we introduce Response Resampling: if a model fails to produce a correct answer in its initial sample group, it is re-engaged until at least one valid response is found. Furthermore, Advantage Reweighting assigns higher priority to "harder" questions—those with a higher ratio of incorrect responses—during gradient updates, ensuring the model focuses on the most challenging defects.

Two-Stage Training

Training proceeds in two phases:

  1. Alignment Stage: The MLLM is frozen while the soft prompt and projector are trained via SFT to establish basic discrimination and localization capabilities.
  2. Full Fine-Tuning Stage: The entire model is optimized using Difficulty-Aware GRPO to achieve cohesive reasoning and high accuracy.

Experimental Results

EMIT was evaluated on the MMAD benchmark, covering seven subtasks including anomaly discrimination, defect classification, and localization. Training data was curated from the Vision, Real-IAD, and MPDD datasets, supplemented by GPT-4 Turbo-generated text descriptions to mitigate the lack of real defective images.

Radar chart comparing EMIT-8B against InternVL3-8B variants across seven industrial inspection tasks.

As shown in the performance analysis, EMIT-8B achieved an average accuracy of 81.95%, representing a 7.77% improvement over the base InternVL3-8B model. Notably, EMIT outperformed standard SFT by 5.03%, with particularly strong gains in Defect Localization (+18.38%) and Defect Classification (+11.69%).

Discussion and Conclusion

A key insight from this work is the effectiveness of using high-quality text as a proxy for visual defects when data is scarce. By leveraging the semantic alignment of MLLMs, EMIT can learn to identify defects it has never seen in image form. Moreover, the identification and resolution of the "Hard Sample" bottleneck in GRPO proves crucial for industrial applications where defects are often subtle.

While EMIT excels at defect-related reasoning, its performance in general object analysis remains a target for future improvement. Future work will explore more complex Chain-of-Thought reasoning chains tailored specifically for industrial logic, such as distinguishing between cosmetic flaws and critical safety failures.

Summary updated

Abstract

Industrial anomaly detection (IAD) plays a crucial role in maintaining the safety and reliability of manufacturing systems. While multimodal large language models (MLLMs) show strong vision-language reasoning abilities, their effectiveness in IAD remains limited without domain-specific adaptation. In this work, we propose EMIT, a unified framework that enhances MLLMs for IAD via difficulty-aware group relative policy optimization (GRPO). EMIT constructs a multi-task IAD dataset and utilizes GPT-generated object text descriptions to compensate for missing defective images. For few-shot anomaly detection, it integrates a soft prompt and heatmap-guided contrastive embeddings derived from patch-level comparisons. To better handle difficult data samples, i.e., cases where the MLLM struggles to generate correct answers, we propose a difficulty-aware GRPO that extends the original GRPO by incorporating a response resampling strategy to ensure the inclusion of correct answers in the sampled responses, as well as an advantage reweighting mechanism to strengthen learning from such difficult data samples. Extensive experiments on the MMAD benchmark demonstrate that EMIT significantly enhances the IAD performance of MLLMs, achieving an average improvement of 7.77\% over the base model (InternVL3-8B) across seven tasks.
BibTeX
@misc{guan2025emit,
  author = {Wei Guan and Jun Lan and Jian Cao and Hao Tan and Huijia Zhu and Weiqiang Wang},
  title = {{EMIT: Enhancing MLLMs for Industrial Anomaly Detection via Difficulty-Aware GRPO}},
  year = {2025},
  eprint = {2507.21619},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.21619},
}
Figure