Abstract
Industrial Anomaly Detection (IAD) is a critical component of manufacturing safety, yet general-purpose Multimodal Large Language Models (MLLMs) often struggle with the domain-specific nuances and data scarcity inherent in industrial settings. We present EMIT, a novel framework designed to bridge this gap. EMIT utilizes a Difficulty-Aware Group Relative Policy Optimization (GRPO) algorithm alongside heatmap-guided contrastive embeddings to enhance the reasoning and localization capabilities of MLLMs. Our approach achieves a significant 7.77% average performance improvement over baseline models across diverse industrial inspection tasks.
Introduction
Traditional IAD systems typically provide binary labels or pixel-level masks, lacking the sophisticated reasoning required for complex industrial diagnostics. While MLLMs like GPT-4V or InternVL possess advanced reasoning capabilities, their application to IAD is hindered by three primary challenges: a significant domain gap between general and industrial imagery, the scarcity of defective samples for supervised fine-tuning (SFT), and alignment failures in standard Reinforcement Learning from Human Feedback (RLHF). Specifically, algorithms like GRPO often fail on "hard" samples where the model cannot initially generate any correct responses, leaving it without a positive learning signal.
Methodology: The EMIT Framework
EMIT addresses these challenges through a combination of architectural innovations and a refined reinforcement learning strategy.
Architectural Enhancements
To handle few-shot anomaly detection, EMIT incorporates Contrastive Embeddings. By comparing a "query" image against a "reference" normal image using a shared Vision Transformer (ViT), the system generates a heatmap based on patch-level cosine distances. This heatmap is then projected into the MLLM’s embedding space, effectively highlighting anomalous regions. Additionally, learnable Soft Prompts are employed to align task-specific queries with these visual contrastive features.
Difficulty-Aware GRPO
The core of EMIT is an evolution of the GRPO algorithm. To overcome the "cold start" problem for difficult industrial samples, we introduce Response Resampling: if a model fails to produce a correct answer in its initial sample group, it is re-engaged until at least one valid response is found. Furthermore, Advantage Reweighting assigns higher priority to "harder" questions—those with a higher ratio of incorrect responses—during gradient updates, ensuring the model focuses on the most challenging defects.
Two-Stage Training
Training proceeds in two phases:
- Alignment Stage: The MLLM is frozen while the soft prompt and projector are trained via SFT to establish basic discrimination and localization capabilities.
- Full Fine-Tuning Stage: The entire model is optimized using Difficulty-Aware GRPO to achieve cohesive reasoning and high accuracy.
Experimental Results
EMIT was evaluated on the MMAD benchmark, covering seven subtasks including anomaly discrimination, defect classification, and localization. Training data was curated from the Vision, Real-IAD, and MPDD datasets, supplemented by GPT-4 Turbo-generated text descriptions to mitigate the lack of real defective images.
As shown in the performance analysis, EMIT-8B achieved an average accuracy of 81.95%, representing a 7.77% improvement over the base InternVL3-8B model. Notably, EMIT outperformed standard SFT by 5.03%, with particularly strong gains in Defect Localization (+18.38%) and Defect Classification (+11.69%).
Discussion and Conclusion
A key insight from this work is the effectiveness of using high-quality text as a proxy for visual defects when data is scarce. By leveraging the semantic alignment of MLLMs, EMIT can learn to identify defects it has never seen in image form. Moreover, the identification and resolution of the "Hard Sample" bottleneck in GRPO proves crucial for industrial applications where defects are often subtle.
While EMIT excels at defect-related reasoning, its performance in general object analysis remains a target for future improvement. Future work will explore more complex Chain-of-Thought reasoning chains tailored specifically for industrial logic, such as distinguishing between cosmetic flaws and critical safety failures.
Summary updated