Coarse-to-Fine Detection of Multiple Seams for Robotic Welding

Pengkun Wei, Shuo Cheng, Dayou Li, Ran Song, Yipeng Zhang, Wei ZhangIROS 2024 · DOI

TL;DR

This paper introduces a novel coarse-to-fine framework for simultaneously detecting multiple weld seams using RGB-D data. By leveraging a vision-language model (FastSAM) for coarse localization and a region-growing algorithm for fine edge extraction, the method achieves sub-millimeter accuracy and significantly improves efficiency for robotic welding applications.

What Problem Does This Paper Address?

Autonomous welding systems require precise identification and localization of weld seams to ensure high-quality welds. Existing methods often focus on detecting single weld seams one by one, which is inefficient for complex workpieces with multiple seams. Furthermore, achieving sub-millimeter accuracy while maintaining efficiency is a significant challenge, especially when dealing with large point cloud datasets from sensors with a wide field of view. This paper aims to address these limitations by proposing a method that can efficiently detect multiple weld seams simultaneously with high accuracy.

Methods

The proposed method employs a two-stage, coarse-to-fine approach:

  • Coarse Seam Detection (Vision-Language Model - VLM): This stage uses the Fast Segment Anything Model (FastSAM), a real-time CNN-based segmentation model, to process RGB images. By providing simple prompts (e.g., clicking on workpiece surfaces), FastSAM segments different surfaces. The intersection areas of these segmented surfaces are used to approximate the location of weld seams. This information is then used to crop the raw point cloud data, focusing only on the regions likely to contain weld seams. This significantly reduces the amount of data to be processed in the subsequent stage.
  • Fine Edge Extraction (Region Growing): The cropped point cloud is then preprocessed, including pass-through filtering and voxel-downsampling to reduce density while preserving geometric information. A region-growing algorithm is applied to segment the workpiece surfaces and identify potential edge points. The algorithm calculates curvature and angle thresholds to distinguish weld seam edges from other surface features. Finally, these precise edge points are used to fit a plane and generate a 6-DOF welding path, including trajectories and torch poses, ensuring collision avoidance and optimal welding angles.

Datasets

The paper does not explicitly mention specific public datasets used for training or evaluation. Instead, experiments were conducted on custom workpieces featuring both linear and curved weld seams, and the method was also deployed in industrial scenarios. The authors mention annotating data for keypoint detection algorithm training, which is part of their automatic prompt generation for FastSAM.

Key Insights and Ideas

  • Leveraging VLMs for Coarse Localization: Using a VLM like FastSAM for initial segmentation of RGB images proves highly effective in distilling relevant information and cropping point clouds, drastically reducing computational load.
  • Coarse-to-Fine Efficiency: The two-stage approach effectively balances the need for broad coverage (coarse stage) with high precision (fine stage), leading to significant efficiency gains.
  • Region Growing for Robust Edge Detection: The region-growing algorithm, guided by curvature and angle analysis, demonstrates robustness in extracting weld seam edges from noisy point cloud data.
  • Simultaneous Multi-Seam Detection: A key advantage is the ability to detect and process multiple weld seams concurrently, a significant improvement over single-seam methods.

Key Results

  • The method achieves sub-millimeter accuracy for weld seam detection.
  • The FastSAM-based cropping module removes over 80% of redundant point clouds and saves approximately 68.5% of computational time when processing a workpiece with ten linear weld seams.
  • Experiments on various workpieces (linear and curved seams) and in industrial scenarios demonstrated promising results and efficiency.
  • Table I shows that the proposed method achieves competitive or superior efficiency (lower computation time per weld) compared to several baseline methods, especially when handling multiple welds. For instance, processing 10 linear welds takes 10.54 seconds of exploration time and 3.05 seconds of computation time per weld, resulting in a total computation time of 11.06 seconds per weld.

Limitations and Future Work

  • Limitations: The paper acknowledges that the accuracy of the method is influenced by the quality of the input point cloud data and the effectiveness of the VLM's segmentation. The current method relies on specific prompts for FastSAM, and while they propose automatic prompt generation, human interaction is still mentioned as the most accurate way to provide prompts.
  • Future Work: The authors plan to extend their work to reconstruct more complex weldments, including those with mixed linear and curved seams, and to extract weld seams from these reconstructed point clouds.

Summary updated

BibTeX
@inproceedings{wei2024coarse,
  author = {Pengkun Wei and Shuo Cheng and Dayou Li and Ran Song and Yipeng Zhang and Wei Zhang},
  title = {{Coarse-to-Fine Detection of Multiple Seams for Robotic Welding}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802546},
}
Figure