Beyond CLIP: Calibrated (Few-Shot) Defect Detection With System One Decision Models

Fadel M. Megahed, Ying-Ju ChenarXiv 2026 · arXiv

TL;DR

This paper introduces a framework for industrial defect detection using "System One" decision models (like Jev and Clef-Flash) that achieves high accuracy with as few as two reference images per class. By combining an agentic AI loop for text-prompt refinement and Platt scaling for probability calibration, the authors outperform standard CLIP-based methods while providing the reliable probability estimates necessary for real-world manufacturing decisions.

What Problem Does This Paper Address?

In the era of Industry 4.0, automated visual inspection is standard, but it faces a massive data bottleneck. Traditional deep learning models require thousands of images. Even modern vision-language models like CLIP, which are touted for "few-shot" capabilities, often still need 50–100 examples per class to perform reliably in niche industrial domains.

For high-yield processes (where defects are rare) or low-volume custom manufacturing, collecting even 50 defective images is often impossible. Furthermore, when CLIP is used, its confidence scores are notoriously poor. As shown in the authors' analysis of previous work, CLIP’s predicted probabilities for both nominal and defective parts often cluster tightly around 0.5, making them useless for risk-based decision-making.

CLIP's probability compression problem: scores cluster near 0.5 regardless of actual class.

Methods

The authors propose a four-stage framework designed to work with System One decision models—a new class of structured models that use input images and text descriptions to answer predefined questions (primitives) like "Choice" or "Score."

  1. Task Definition: The user provides a question, basic text descriptions for "nominal" and "defective" states, and a tiny set of reference images ($k=2$).
  2. Agentic Description Refinement: Because manual prompt engineering is tedious and model-specific, the authors use an "agentic AI loop" (powered by Claude). The agent iteratively rewrites the text descriptions, testing them against a small development set to maximize the Area Under the Curve (AUC).
  3. Probability Calibration: To fix the "uncertainty" problem, the framework applies Platt scaling. This transforms raw model scores into calibrated probabilities that actually reflect the likelihood of a defect.
  4. Inference: The final, fixed setup (refined text + 2 reference images + calibration function) is used to inspect new parts in real-time.
FIGURE 2: Overview of our k−shot learning framework for using decision models (e.g., Jev and Clef-Flash) to inspect manufactured parts.

Datasets

The framework was evaluated across five distinct manufacturing case studies:

  • Metallic pans: Surface nonconformities.
  • 3D printing extrusion: Over- and under-extrusion profiles.
  • Textiles: Local and global disturbances in stochastic textures.
  • Pipe staples: Missing components in automotive assemblies.
  • Microstructure: Simulated anomalies in material grain structures.
FIGURE 3: An example of the two nominal and two defective refer- ence images per manufacturing case study, used by the decision models and reference-matched CLIP.

Key Insights and Ideas

  • System One > General Purpose: While CLIP is a powerful generalist, structured decision models like Jev are better at "triangulating" a defect when given both a textual description and a visual reference.
  • The Power of the Agentic Loop: The AI agent often discovered descriptive nuances that humans might miss. For extrusion, it moved from a generic "visible manufacturing defect" to specific mentions of "material width," "continuity," and "dark gaps," significantly boosting AUC from 0.59 to 0.78.
  • Calibration is Mandatory: High accuracy isn't enough for a factory floor. If a model says there is a 90% chance of a defect, that should mean 9 out of 10 such parts are actually defective. The authors show that without calibration, even high-accuracy models provide misleading confidence scores.

Key Results

  • Superior Accuracy with Less Data: Using only $k=2$ reference images, Jev achieved accuracies between 0.75 and 1.00, and Clef-Flash between 0.71 and 0.99. Both consistently outperformed CLIP matched with the same number of references.
  • Beating the Baseline: In several cases (like metallic pans and pipe staples), Jev with just 2 images outperformed a CLIP baseline that used 50 images.
  • Refinement Works: Agentic refinement improved accuracy in 6 out of 8 model-application combinations. The most dramatic jump was for Clef-Flash on pipe staples, where accuracy rose by 0.53.
  • Reliability: Platt scaling reduced the Brier score (a measure of probability accuracy) in 13 of the 15 tested configurations, making the models far more trustworthy for automated rejection systems.
FIGURE 4: Paired accuracy differences (95% bootstrap confidence intervals) for Jev and Clef-Flash: (a) versus CLIP, (b) refined versus generic descriptions, and (c) two versus zero reference images per class.

Limitations and Future Work

The authors acknowledge that their tests were conducted under relatively stable lab conditions; real-world factory floors with shifting lighting and dust might challenge these models. Currently, the framework is optimized for binary classification (defect vs. no defect). Future work aims to extend this to multi-class problems (identifying which defect) and defect localization (finding where the defect is) using newly released "Box" primitives in System One models.

Finally, while the agentic loop improves performance, the models remain "black boxes." Understanding why specific text descriptions work better than others remains an open research question.

Summary updated

Abstract

Cameras and vision sensors are widely used for product and process inspection in manufacturing environments. Traditionally, this process required thousands of images of both nominal and defective products to train such models. Recent research shows that vision transformer (ViT) models, such as CLIP, can reduce this requirement to 50-100 images per nominal and defective class. However, collecting even 50 defective images can be difficult in high-yield or low-volume manufacturing processes. In this paper, we examine the utility of two recently proposed structured decision models (Jev and Clef-Flash) in applications using as few as k=2 reference images per class at inference. In the process, we provide three main contributions: (a) a framework for k-shot learning of these models for image classification/inspection applications; (b) an agentic approach for automatically refining the text descriptions that guide image classification in these decision models; and (c) an approach to calibrate the resulting defect probabilities to improve their reliability for inspection decisions. We test these contributions in five manufacturing case studies. Using two reference images per class, both Jev and Clef-Flash outperform CLIP's test accuracies with the same references (0.75-1.00 [Jev] and 0.71-0.99 [Clef-Flash] vs 0.67-0.84 [CLIP]). Furthermore, we show that our agentic refinement of text descriptions significantly improves accuracy across 6 of 8 model-application combinations, and our calibration approach reduces the Brier score in 13 of the 15 model-application configurations. To support the reproducibility of our work, we provide our replication code and have created the R "decisions" package for evaluating additional models. We also provide a publicly available web app that allows practitioners to inspect their own images with no code.
BibTeX
@misc{megahed2026beyond,
  author = {Fadel M. Megahed and Ying-Ju Chen},
  title = {{Beyond CLIP: Calibrated (Few-Shot) Defect Detection With System One Decision Models}},
  year = {2026},
  eprint = {2610.41199},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41199},
}
Figure