TL;DR
This paper investigates how well Multimodal Large Language Models (MLLMs) can assess the quality of welds, a critical task in manufacturing. They found that while MLLMs perform better on online images, they can still generalize to real-world welds, and a new prompting strategy called WeldPrompt shows promise in improving their performance.
What Problem Does This Paper Address?
The increasing adoption of AI, particularly Large Language Models (LLMs), has largely focused on white-collar jobs. However, skilled production work, like welding, presents a unique challenge and opportunity for multimodal LLMs that can process both text and images. A key problem is understanding how well these models can perform in high-stakes, technical domains where errors can have significant consequences. Specifically, the paper addresses the gap in research concerning MLLMs' ability to understand and evaluate the quality of physical objects in real-world industrial settings, moving beyond simple object recognition to nuanced assessment.
Methods
The researchers evaluated two state-of-the-art MLLMs: GPT-40 and LLaVA-1.6. They employed two prompting strategies:
- Zero-Shot Prompting: The model receives only the weld image and is asked to provide a binary assessment (acceptable/unacceptable) along with a brief explanation.
- WeldPrompt: This novel strategy combines Chain-of-Thought (CoT) generation with in-context learning. It provides the model with access to similar welds that were correctly assessed by an expert, along with their reasoning, to guide its own assessment. This aims to mitigate hallucinations and improve reasoning by leveraging examples and step-by-step thinking.
Datasets
The study utilized two datasets of weld images:
- Real-World Weld Dataset: Collected from expert welders in manufacturing environments, these images represent actual production scenarios and were annotated by a welding expert.
- Online Weld Dataset: Assembled from targeted Google image searches using keywords related to welding problems and defects, this dataset provides a broader range of examples, potentially including those seen during model training.
Both datasets were annotated by a welding expert for acceptability across three contexts: RV & Marine, Aeronautical, and Farming, reflecting diverse requirements and tolerances.
Key Insights and Ideas
- Online vs. Real-World Performance: MLLMs consistently perform better on online weld images compared to unseen, real-world images. This suggests a reliance on memorization of training data rather than deep reasoning about novel, real-world scenarios.
- Generalization Capability: Despite the performance gap, the models still perform "reasonably well" on real-world images, indicating a growing ability to handle unfamiliar data, which is a positive sign for industrial applications.
- WeldPrompt's Potential: The WeldPrompt strategy shows promise in improving model performance, particularly in enhancing recall and aligning model predictions with expert judgments in certain contexts.
- Context Matters: The performance of MLLMs varies significantly across different welding contexts (RV & Marine, Aeronautical, Farming), highlighting the need for domain-specific fine-tuning and data.
- Trade-offs in Prompting: WeldPrompt can lead to trade-offs, such as increased precision at the cost of recall, or vice-versa, depending on the model and context.
Key Results
- Zero-Shot Performance: GPT-40 generally outperformed LLaVA-1.6 across all contexts and datasets. Both models performed better on the Online dataset than the Real-World dataset.
- WeldPrompt Impact: In the RV & Marine context, WeldPrompt improved both recall and precision for GPT-40. However, in the Aeronautical context, GPT-40 became more conservative, rejecting more welds. For LLaVA-1.6, WeldPrompt showed a slight decrease in F1 scores compared to zero-shot.
- Real-World Challenges: LLaVA-1.6 showed zero classification performance on the Real-World Aeronautical dataset, likely due to class imbalance. GPT-40 was less affected by imbalance and predicted welds as acceptable more aggressively.
- Overall Averages: Applying WeldPrompt led to a slight decrease in average performance on the Real-World dataset but an increase on the Online dataset, reinforcing the idea of reliance on memorization.
Limitations and Future Work
- Memorization vs. Reasoning: The study strongly suggests that current MLLMs rely heavily on memorization rather than genuine reasoning when classifying welding images, especially for real-world data.
- Domain-Specific Reasoning: MLLMs struggle with nuanced, technical, image-based reasoning in domains like welding, indicating a need for further development in this area.
- Fine-tuning and Data: The authors emphasize the importance of domain-specific data and fine-tuning to improve model reliability in high-stakes industrial applications.
- WeldPrompt's Inconsistency: WeldPrompt's performance was inconsistent across different contexts and models, suggesting that further refinement is needed.
- Future Research: Future work should focus on improving MLLMs' ability to reason in unfamiliar domains, potentially through retrieval-augmented methods, in-context learning, or domain-specific fine-tuning. The integration of Explainable AI (XAI) is also crucial for making these systems more interpretable and aligned with human decision-making. The study also suggests that task-specific, fine-tuned models might offer more reliable solutions than large, general-purpose MLLMs for production deployments.
Summary updated