Abstract
Industrial defect detection remains a significant challenge for general-purpose Large Vision-Language Models (LVLMs) due to the domain gap between natural and industrial imagery, the inefficiency of manual prompting, and a lack of specialized large-scale data. This paper introduces the Multi-Modal Industrial Open Dataset (MMIO-80K), the first large-scale multi-modal benchmark for industrial zero-shot learning, containing over 80,000 samples. Alongside this dataset, we propose the Refined Text-Visual Prompt (RTVP) framework. RTVP automates the prompting process through an expert-guided adaptation of Mobile-SAM, eliminating the need for human intervention while achieving state-of-the-art performance in identifying unseen industrial defects.
Introduction
While models like the Segment Anything Model (SAM) have set new standards in computer vision, their application in industrial settings is hindered by three primary factors. First, the domain gap: industrial images feature unique lighting and sparse features that differ from the natural datasets used for pre-training. Second, prompt subjectivity: existing models often rely on manual human input (clicks or boxes), which is a bottleneck in high-speed manufacturing. Third, data scarcity: there has been a lack of unified, multi-modal datasets for industrial zero-shot learning.
The MMIO-80K Dataset
To address the data gap, we present the MMIO-80K dataset. This benchmark includes over 80,000 samples across 6 super-categories (metallurgy, automobile, electronics, textiles, daily necessities, and wood) and 18 subcategories. It features 100 distinct attributes for fine-grained detection and includes a custom-collected subset of high-resolution glass container images.
The RTVP Framework
The proposed Refined Text-Visual Prompt (RTVP) framework introduces an automated, "expert-guided" approach to industrial segmentation. Unlike traditional methods that require manual prompts, RTVP generates them automatically and refines them through cross-modal interaction.
Methodology Components
- Expert-assisted Domain Adaptation: A specialized CNN "expert" trained on industrial data supervises the adaptation of Mobile-SAM. This allows the larger model to converge more efficiently on industrial features.
- Refined Visual Prompter: The system uses a sparse modeling sample selection strategy to identify high-frequency areas where defects typically occur, automatically picking the most informative pixels based on uncertainty scores.
- Cross-modal Prompt Interaction: Text and visual prompts are integrated, allowing text embeddings to refine visual features. This ensures the model understands how a visual anomaly relates to a specific textual description, such as an "oil spot."
Key Insights and Results
Our evaluation reveals that automated prompting significantly outperforms manual methods by eliminating subjective noise. Furthermore, we found that sparse sampling—focusing on specific high-frequency pixels—is more effective for detecting the tiny, sparse defects common in industrial scenarios than the dense prompting used for natural scenes.
Performance Metrics
- Zero-Shot Detection: On the MMIO-80K zero-shot task, RTVP achieved 24.7% AP, outperforming Grounding DINO (20.2%) and YOLO-World (21.2%).
- Closed-Set Detection: In standard tasks, it reached 42.4% AP, surpassing specialized models like YOLOv10.
- Generalization: RTVP also improved performance on the natural-scene MS-COCO dataset (47.2% AP), demonstrating the robustness of its refined prompting mechanism.
Conclusion and Future Work
RTVP and the MMIO-80K dataset establish a new baseline for industrial defect detection. While an AP of 24.7% in zero-shot scenarios highlights the difficulty of the task, the removal of the "human-in-the-loop" represents a major step toward practical industrial deployment. Future research will focus on expanding the dataset to more niche categories and optimizing cross-modal interactions for complex, multi-attribute defects.
Summary updated