Zero-Shot Learning in Industrial Scenarios: New Large-Scale Benchmark, Challenges and Baseline

Zekai Zhang, Qinghui Chen, Maomao Xiong, Shijiao Ding, Zhanzhi Su, Xinjie Yao, Yiming Sun, Cong Bai, Jinglin Zhang

Abstract

Industrial defect detection remains a significant challenge for general-purpose Large Vision-Language Models (LVLMs) due to the domain gap between natural and industrial imagery, the inefficiency of manual prompting, and a lack of specialized large-scale data. This paper introduces the Multi-Modal Industrial Open Dataset (MMIO-80K), the first large-scale multi-modal benchmark for industrial zero-shot learning, containing over 80,000 samples. Alongside this dataset, we propose the Refined Text-Visual Prompt (RTVP) framework. RTVP automates the prompting process through an expert-guided adaptation of Mobile-SAM, eliminating the need for human intervention while achieving state-of-the-art performance in identifying unseen industrial defects.

Introduction

While models like the Segment Anything Model (SAM) have set new standards in computer vision, their application in industrial settings is hindered by three primary factors. First, the domain gap: industrial images feature unique lighting and sparse features that differ from the natural datasets used for pre-training. Second, prompt subjectivity: existing models often rely on manual human input (clicks or boxes), which is a bottleneck in high-speed manufacturing. Third, data scarcity: there has been a lack of unified, multi-modal datasets for industrial zero-shot learning.

The MMIO-80K Dataset

To address the data gap, we present the MMIO-80K dataset. This benchmark includes over 80,000 samples across 6 super-categories (metallurgy, automobile, electronics, textiles, daily necessities, and wood) and 18 subcategories. It features 100 distinct attributes for fine-grained detection and includes a custom-collected subset of high-resolution glass container images.

Visual samples from the MMIO-80K dataset, showcasing the diversity of industrial defect categories.

The RTVP Framework

The proposed Refined Text-Visual Prompt (RTVP) framework introduces an automated, "expert-guided" approach to industrial segmentation. Unlike traditional methods that require manual prompts, RTVP generates them automatically and refines them through cross-modal interaction.

Methodology Components

  • Expert-assisted Domain Adaptation: A specialized CNN "expert" trained on industrial data supervises the adaptation of Mobile-SAM. This allows the larger model to converge more efficiently on industrial features.
  • Refined Visual Prompter: The system uses a sparse modeling sample selection strategy to identify high-frequency areas where defects typically occur, automatically picking the most informative pixels based on uncertainty scores.
  • Cross-modal Prompt Interaction: Text and visual prompts are integrated, allowing text embeddings to refine visual features. This ensures the model understands how a visual anomaly relates to a specific textual description, such as an "oil spot."
Comparison between traditional SAM prompting and the proposed RTVP framework.

Key Insights and Results

Our evaluation reveals that automated prompting significantly outperforms manual methods by eliminating subjective noise. Furthermore, we found that sparse sampling—focusing on specific high-frequency pixels—is more effective for detecting the tiny, sparse defects common in industrial scenarios than the dense prompting used for natural scenes.

Performance Metrics

  • Zero-Shot Detection: On the MMIO-80K zero-shot task, RTVP achieved 24.7% AP, outperforming Grounding DINO (20.2%) and YOLO-World (21.2%).
  • Closed-Set Detection: In standard tasks, it reached 42.4% AP, surpassing specialized models like YOLOv10.
  • Generalization: RTVP also improved performance on the natural-scene MS-COCO dataset (47.2% AP), demonstrating the robustness of its refined prompting mechanism.

Conclusion and Future Work

RTVP and the MMIO-80K dataset establish a new baseline for industrial defect detection. While an AP of 24.7% in zero-shot scenarios highlights the difficulty of the task, the removal of the "human-in-the-loop" represents a major step toward practical industrial deployment. Future research will focus on expanding the dataset to more niche categories and optimizing cross-modal interactions for complex, multi-attribute defects.

Summary updated

Abstract

Large Visual Language Models (LVLMs) have achieved remarkable success in vision tasks. However, the significant differences between industrial and natural scenes make applying LVLMs challenging. Existing LVLMs rely on user-provided prompts to segment objects. This often leads to suboptimal performance due to the inclusion of irrelevant pixels. In addition, the scarcity of data also makes the application of LVLMs in industrial scenarios remain unexplored. To fill this gap, this paper proposes an open industrial dataset and a Refined Text-Visual Prompt (RTVP) for zero-shot industrial defect detection. First, this paper constructs the Multi-Modal Industrial Open Dataset (MMIO) containing 80K+ samples. MMIO contains diverse industrial categories, including 6 super categories and 18 subcategories. MMIO is the first large-scale multi-scenes pre-training dataset for industrial zero-shot learning, and provides valuable training data for open models in future industrial scenarios. Based on MMIO, this paper provides a RTVP specifically for industrial zero-shot tasks. RTVP has two significant advantages: First, this paper designs an expert-guided large model domain adaptation mechanism and designs an industrial zero-shot method based on Mobile-SAM, which enhances the generalization ability of large models in industrial scenarios. Second, RTVP automatically generates visual prompts directly from images and considers text-visual prompt interactions ignored by previous LVLM, improving visual and textual content understanding. RTVP achieves SOTA with 42.2% and 24.7% AP in zero-shot and closed scenes of MMIO.
BibTeX
@inproceedings{zhang2025zero,
  author = {Zekai Zhang and Qinghui Chen and Maomao Xiong and Shijiao Ding and Zhanzhi Su and Xinjie Yao and Yiming Sun and Cong Bai and Jinglin Zhang},
  title = {{Zero-Shot Learning in Industrial Scenarios: New Large-Scale Benchmark, Challenges and Baseline}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i10.33124},
}

Loading related content...

Figure