TL;DR
This paper introduces a framework for industrial defect detection using "System One" decision models (like Jev and Clef-Flash) that achieves high accuracy with as few as two reference images per class. By combining an agentic AI loop for text-prompt refinement and Platt scaling for probability calibration, the authors outperform standard CLIP-based methods while providing the reliable probability estimates necessary for real-world manufacturing decisions.
What Problem Does This Paper Address?
In the era of Industry 4.0, automated visual inspection is standard, but it faces a massive data bottleneck. Traditional deep learning models require thousands of images. Even modern vision-language models like CLIP, which are touted for "few-shot" capabilities, often still need 50–100 examples per class to perform reliably in niche industrial domains.
For high-yield processes (where defects are rare) or low-volume custom manufacturing, collecting even 50 defective images is often impossible. Furthermore, when CLIP is used, its confidence scores are notoriously poor. As shown in the authors' analysis of previous work, CLIP’s predicted probabilities for both nominal and defective parts often cluster tightly around 0.5, making them useless for risk-based decision-making.
Methods
The authors propose a four-stage framework designed to work with System One decision models—a new class of structured models that use input images and text descriptions to answer predefined questions (primitives) like "Choice" or "Score."
- Task Definition: The user provides a question, basic text descriptions for "nominal" and "defective" states, and a tiny set of reference images ($k=2$).
- Agentic Description Refinement: Because manual prompt engineering is tedious and model-specific, the authors use an "agentic AI loop" (powered by Claude). The agent iteratively rewrites the text descriptions, testing them against a small development set to maximize the Area Under the Curve (AUC).
- Probability Calibration: To fix the "uncertainty" problem, the framework applies Platt scaling. This transforms raw model scores into calibrated probabilities that actually reflect the likelihood of a defect.
- Inference: The final, fixed setup (refined text + 2 reference images + calibration function) is used to inspect new parts in real-time.
Datasets
The framework was evaluated across five distinct manufacturing case studies:
- Metallic pans: Surface nonconformities.
- 3D printing extrusion: Over- and under-extrusion profiles.
- Textiles: Local and global disturbances in stochastic textures.
- Pipe staples: Missing components in automotive assemblies.
- Microstructure: Simulated anomalies in material grain structures.
Key Insights and Ideas
- System One > General Purpose: While CLIP is a powerful generalist, structured decision models like Jev are better at "triangulating" a defect when given both a textual description and a visual reference.
- The Power of the Agentic Loop: The AI agent often discovered descriptive nuances that humans might miss. For extrusion, it moved from a generic "visible manufacturing defect" to specific mentions of "material width," "continuity," and "dark gaps," significantly boosting AUC from 0.59 to 0.78.
- Calibration is Mandatory: High accuracy isn't enough for a factory floor. If a model says there is a 90% chance of a defect, that should mean 9 out of 10 such parts are actually defective. The authors show that without calibration, even high-accuracy models provide misleading confidence scores.
Key Results
- Superior Accuracy with Less Data: Using only $k=2$ reference images, Jev achieved accuracies between 0.75 and 1.00, and Clef-Flash between 0.71 and 0.99. Both consistently outperformed CLIP matched with the same number of references.
- Beating the Baseline: In several cases (like metallic pans and pipe staples), Jev with just 2 images outperformed a CLIP baseline that used 50 images.
- Refinement Works: Agentic refinement improved accuracy in 6 out of 8 model-application combinations. The most dramatic jump was for Clef-Flash on pipe staples, where accuracy rose by 0.53.
- Reliability: Platt scaling reduced the Brier score (a measure of probability accuracy) in 13 of the 15 tested configurations, making the models far more trustworthy for automated rejection systems.
Limitations and Future Work
The authors acknowledge that their tests were conducted under relatively stable lab conditions; real-world factory floors with shifting lighting and dust might challenge these models. Currently, the framework is optimized for binary classification (defect vs. no defect). Future work aims to extend this to multi-class problems (identifying which defect) and defect localization (finding where the defect is) using newly released "Box" primitives in System One models.
Finally, while the agentic loop improves performance, the models remain "black boxes." Understanding why specific text descriptions work better than others remains an open research question.
Summary updated