TL;DR
This paper introduces "Echo," a novel multi-expert framework designed to enhance the performance of Multimodal Large Language Models (MLLMs) for industrial anomaly detection. Echo integrates specialized modules for knowledge retrieval, reference image comparison, structured reasoning, and decision-making, enabling MLLMs to achieve greater accuracy, adaptability, and robustness in identifying defects in industrial settings.
What Problem Does This Paper Address?
Industrial anomaly detection (IAD) is critical for ensuring product quality and operational safety. Traditional deep learning models for IAD often struggle with flexibility and adaptability, requiring extensive retraining for new defect types or production environments. While recent Multimodal Large Language Models (MLLMs) show promise by combining visual and textual understanding, they lack crucial domain-specific knowledge, such as industry-specific defect tolerance levels. This deficiency can lead to inaccurate or "hallucinated" outputs when MLLMs are directly applied to IAD tasks. The paper aims to bridge this gap by guiding MLLMs with specialized expertise to improve their performance in complex industrial anomaly detection scenarios.
Methods
The core contribution is the "Echo" framework, a multi-expert system designed to augment MLLMs for IAD. Echo comprises four key expert modules:
- Reference Extractor: This module retrieves the most similar normal image to a given query image. This serves as a contextual baseline for anomaly comparison and helps establish relevant anomaly tolerance levels.
- Knowledge Guide: This module provides task-specific, domain knowledge crucial for understanding industry-relevant details and defect tolerances. It leverages information from datasets like MMAD and is enhanced by a "Context-Specific Relevance Extractor" that tailors the knowledge to the specific query.
- Reasoning Expert: This module enables structured, stepwise logical reasoning to tackle complex queries that require multi-step analysis, mimicking the cognitive processes of a skilled inspector.
- Decision Maker: This module synthesizes insights from all other expert modules to produce a precise, context-aware, and definitive response.
The framework operates by first classifying the query type to determine which expert modules are needed. These selected modules then process the query image and text, and their combined outputs are fed to the MLLM for final decision generation. The paper also highlights the use of the HNSW algorithm for efficient multimodal retrieval from a large memory of embeddings.
Datasets
The experiments were conducted using two datasets from the MMAD benchmark:
- MVTec-AD: Contains 15 categories of industrial objects.
- VisA: Contains 12 categories of industrial objects.
The MMAD benchmark itself is noted as the first comprehensive benchmark for IAD using MLLMs, evaluating various anomaly types and object classifications.
Key Insights and Ideas
- MLLMs need guidance for specialized tasks: Directly feeding industrial anomaly detection queries to general-purpose MLLMs often results in inaccurate or hallucinated outputs due to a lack of domain-specific knowledge.
- Expert modules are crucial for bridging the gap: Echo's modular approach, with specialized experts for reference extraction, knowledge integration, and reasoning, significantly enhances MLLM performance in IAD.
- Context-specific knowledge is more effective than broad domain knowledge: Tailoring knowledge to the specific query (context-specific) yields better results than simply providing general domain information.
- Reference normal samples are highly beneficial, but with caveats: A single, well-chosen normal reference sample significantly improves anomaly detection, especially for smaller models. However, too many reference samples can sometimes lead to confusion or "hallucinations."
- Larger MLLMs benefit more from advanced retrieval: Larger models (like Qwen2-VL 7B) are better at leveraging multimodal retrieval and contextual information compared to smaller models (like Qwen2-VL 2B).
Key Results
- Echo, when integrated with Qwen2-VL 7B, achieved state-of-the-art performance on the MVTec-AD and VisA datasets across multiple IAD tasks (Anomaly Discrimination, Defect Classification, Defect Localization, Defect Description, and Defect Analysis).
- The integration of Echo provided consistent improvements, with gains of up to +7.39% in Discrimination and +4.40% in Classification on MVTec-AD compared to the baseline Qwen2-VL.
- Echo achieved an average performance increase of +2.58% across all tasks on both datasets compared to other large-scale models.
- The Reference Extractor module, particularly when using retrieved similar normal images (1-shot* setting), showed significant improvements, especially for larger models like Qwen2-VL 7B, with anomaly detection accuracy increasing by +5.99% over the 0-shot setting.
- The multiple-choice question format was found to be the most effective for anomaly discrimination accuracy compared to Q&A or True/False formats.
Limitations and Future Work
- Low-quality images: The current evaluation uses high-quality datasets. Real-world industrial images often suffer from low resolution, noise, or poor lighting, which requires further investigation for MLLM handling.
- Model size and edge deployment: For industrial settings with limited computational resources, focusing on smaller models (like the 2B parameter range) for practical edge deployment is crucial.
- Novel defect types: The current evaluation is based on datasets with predefined defect information. The ability of the framework and MLLMs to detect or reason about entirely novel defect types remains an open question and requires further research.
- Effectiveness of multiple reference samples: While single reference samples are beneficial, the paper notes that including more than one can sometimes lead to diminishing returns or incorrect judgments, suggesting a need for further optimization in handling multiple visual inputs.
Summary updated