Welding Defect Monitoring Based on Multi-Scale Feature Fusion of Molten Pool Videos

Chenbo Shi, Lei Wang, Changsheng Zhu, Tengyue Han, Xiangyu Zhang, Delin Wang, C. ZhangSensors 2024 · DOI

TL;DR

This paper introduces MFVNet, a novel deep learning model for real-time welding defect detection using molten pool videos. By fusing multi-scale spatial features with temporal information from video sequences, MFVNet achieves high accuracy (97.416%) and impressive speed (16 ms per sample), crucial for intelligent automated welding.

What Problem Does This Paper Address?

Automated welding is increasingly vital in manufacturing, but ensuring high-quality welds remains a challenge. A critical aspect of weld quality is monitoring the molten pool, the dynamic liquid metal pool formed during welding. Detecting defects like porosity in real-time is essential for immediate process adjustments. However, existing methods struggle with the inherent dynamism of the molten pool, variations in camera perspectives, and changing pool shapes, making single-frame image analysis insufficient. This paper aims to bridge this gap by developing a robust and efficient system that leverages the temporal and multi-scale spatial characteristics of molten pool videos for accurate defect detection.

Methods

The core of this work is the MFVNet (Multi-Scale Feature Fusion Network), a video-based model designed for efficient welding defect classification. Key methodological contributions include:

  • Lightweight Backbone: The model utilizes MobileNetV2 as its backbone for efficient feature extraction, balancing performance with computational cost.
  • Temporal Shift Modules (TSM): To capture the dynamic nature of the molten pool, TSMs are integrated into each layer of the backbone. These modules efficiently shift features across temporal dimensions, enabling the model to learn from temporal dependencies without significant computational overhead.
  • Multi-Scale Feature Fusion (MFF) Module: This novel module is designed to handle variations in molten pool size and position. It employs three branches with different kernel sizes to extract features at various scales and directions, mimicking the concept of feature pyramids. This allows the model to capture both fine details and broader contextual information.
  • Attention Module (AM): An attention mechanism, incorporating CBAM (Convolutional Block Attention Module) and mixed attention with row/column features, is used to enhance feature processing. This helps the model focus on critical regions and important features within the molten pool, improving its ability to distinguish between normal conditions and defects.
  • Gated Fusion: A gating mechanism is implemented to facilitate effective feature propagation and fusion, allowing the model to prioritize relevant information.

Datasets

  • WELDPOOL Dataset: A custom dataset was created for this study, consisting of molten pool videos collected using a trackless crawling welding robot and a specialized molten pool camera. The dataset includes approximately 600 normal molten pool samples and 600 porosity defect samples, with each sample comprising 8 frames.
  • UCF101-24 and JHMDB: These standard action recognition datasets were used to evaluate the generalization capability of the proposed MFVNet model.

Key Insights and Ideas

  • Temporal Dynamics are Crucial: The study strongly emphasizes that relying solely on static, single-frame images is insufficient for robust molten pool defect detection. The dynamic changes in light spots and pool shape over time provide critical information that MFVNet effectively leverages.
  • Multi-Scale Fusion for Robustness: The MFF module's ability to capture features at different scales is key to handling the inherent variability in molten pool appearance due to camera angles, lighting, and pool size. This makes the model more adaptable to real-world welding scenarios.
  • Attention for Precision: The integration of attention mechanisms significantly improves the model's ability to pinpoint critical features and suppress irrelevant background noise, leading to more accurate defect identification.
  • Efficiency is Paramount: The paper highlights the trade-off between accuracy and real-time performance. By using a lightweight backbone (MobileNetV2) and efficient temporal modules (TSM), MFVNet achieves both high accuracy and low latency, making it suitable for practical industrial applications.

Key Results

  • High Accuracy: MFVNet achieved a top-1 accuracy of 97.416% on the custom WELDPOOL dataset.
  • Real-time Performance: The model boasts a processing time of 16 ms per sample, enabling real-time monitoring. This translates to a throughput of 62.5 V/s.
  • Module Effectiveness: Ablation studies showed that integrating both the MFF and AM modules significantly improved performance over baseline models. For instance, MFVNet achieved a 1.55% improvement in Top-1 accuracy compared to a TSM baseline.
  • Generalization: The model demonstrated good performance on the UCF101-24 dataset, indicating its potential for broader video understanding tasks.

Limitations and Future Work

  • Focus on Porosity: The current model is primarily designed to detect porosity defects. The authors suggest future work should focus on extending the model to identify a wider range of welding defects.
  • Dataset Size: While the WELDPOOL dataset contains 1200 samples, deep learning models generally benefit from larger datasets for enhanced generalization and robustness. Future research could involve collecting more diverse and extensive datasets.
  • Spatter Interference: Tests revealed that spatter can negatively impact defect detection accuracy. Improving the model's robustness to spatter interference is identified as a critical direction for further optimization.

Summary updated

Abstract

Real-time quality monitoring through molten pool images is a critical focus in researching high-quality, intelligent automated welding. However, challenges such as the dynamic nature of the molten pool, changes in camera perspective, and variations in pool shape make defect detection using single-frame images difficult. We propose a multi-scale fusion method for defect monitoring based on molten pool videos to address these issues. This method analyzes the temporal changes in light spots on the molten pool surface, transferring features between frames to capture dynamic behavior. Our approach employs multi-scale feature fusion using row and column convolutions along with a gated fusion module to accommodate variations in pool size and position, enabling the detection of light spot changes of different sizes and directions from coarse to fine. Additionally, incorporating mixed attention with row and column features enables the model to capture the characteristics of the molten pool more efficiently. Our method achieves an accuracy of 97.416% on a molten pool video dataset, with a processing time of 16 ms per sample. Experimental results on the UCF101-24 and JHMDB datasets also demonstrate the method's generalization capability.
BibTeX
@inproceedings{shi2024welding,
  author = {Chenbo Shi and Lei Wang and Changsheng Zhu and Tengyue Han and Xiangyu Zhang and Delin Wang and C. Zhang},
  title = {{Welding Defect Monitoring Based on Multi-Scale Feature Fusion of Molten Pool Videos}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24206561},
}
Figure