Sim2real Transfer Learning for Point Cloud Segmentation: An Industrial Application Case on Autonomous Disassembly

Chengzhi Wu, Xuelei Bi, Julius Pfrommer, Alexander Cebulla, Simon Mangold, Jürgen Beyerer

TL;DR

This paper tackles the challenge of applying deep learning for industrial robotics tasks like autonomous disassembly, where real-world data is scarce and annotation is difficult. They propose a sim2real transfer learning pipeline for point cloud segmentation, successfully adapting models trained on synthetic data to real-world scenarios for precise screw localization, crucial for disassembly.

What Problem Does This Paper Address?

Industrial robotics, particularly in manufacturing and remanufacturing, is increasingly leveraging deep learning (DL) for tasks like object manipulation and disassembly. However, training robust DL models requires vast amounts of annotated data. For robots interacting with the real world, collecting and annotating such data is often prohibitively difficult, time-consuming, and even unsafe. This is especially true for tasks involving precise localization of small components like screws, which are critical for disassembly processes. Existing sim2real approaches primarily focus on image data, leaving a gap in applying these techniques to 3D point cloud data, which offers richer geometric information.

Methods

The core of this work is a sim2real transfer learning pipeline for point cloud segmentation. The process involves:

  • Synthetic Data Generation:
    • Mesh Generation: A custom Blender addon is used to create parametric 3D mesh models of various motor components (pole pot, gear container, screws, etc.). This allows for generating diverse motor variants with controlled parameters.
  • Point Cloud Generation: Using the generated meshes, synthetic point clouds are created with a tool called BlenSor. This process also generates corresponding segmentation ground truth labels.
  • Data Pre-processing and Augmentation:
    • Cropping: To handle large point clouds and focus on relevant areas, a cuboid region around the clamped motor is cropped. This significantly reduces the data size while retaining essential information.
  • Augmentation: Various augmentation techniques are applied to the synthetic data to improve generalization: random rotation/jittering of point clouds, random cuboid size, mild translation/rotation of motors, and adding scene masks.
  • Sim2real Transfer Learning:
    • Pre-training: A DGCNN (Dynamic Graph CNN) backbone with a Spatial Transform Network (STN) is pre-trained on the generated synthetic point cloud dataset. This step learns general features from simulated data.
  • Fine-tuning: The pre-trained model is then fine-tuned on a smaller, manually labeled real-world point cloud dataset. This step adapts the model to the specific characteristics of real-world data.
  • Post-processing for Screw Information: After segmentation, clustering algorithms (DBSCAN) are used to group segmented screw points, identify screw locations, and estimate their orientations based on the cover's orientation.
  • Imbalanced Learning Strategies: To address the inherent imbalance between background and tail categories (like screws), the paper explores:
  • Focused Sampling: A strategy to increase the density of tail category points.
  • Weighting Category Loss: A novel method to assign weights to categories based on their rarity.
  • Patch-based Attention Network: A novel network module that uses an additional kernel loss to encourage learning of features around object boundaries and applies cross-attention between patch-wise and point-wise features.

Datasets

  • Synthetic Dataset: A custom dataset generated using Blender and BlenSor, containing simulated motor models and their corresponding point clouds with segmentation labels. The paper mentions generating 1000 random scenes with 1000 random motors for pre-training.
  • Real-world Dataset: A dataset of 26 real-world point clouds captured with a Zivid camera, manually labeled for segmentation.

Key Insights and Ideas

  • Point Clouds for Sim2real: Applying sim2real to point cloud data is feasible and beneficial for industrial robotics, offering richer geometric information than images alone and avoiding issues with rendering realistic textures.
  • Importance of Data Augmentation: While pre-training on synthetic data is crucial, effective augmentation strategies (especially random cuboid size) significantly improve the model's performance when transferred to real-world data.
  • Addressing Imbalanced Data: The extreme class imbalance in point clouds (background vs. screws) is a significant challenge. While cropping helps, dedicated strategies like category weighting and the proposed patch-based attention network are vital for good performance on tail categories.
  • Patch-based Attention for Boundaries: The novel patch-based attention module effectively learns features around object boundaries, which is particularly useful for segmenting small, critical components like screws.
  • STN for Robustness: Incorporating a Spatial Transform Network (STN) significantly improves segmentation performance by handling camera perspective variations.

Key Results

  • The proposed sim2real pipeline (pre-training + fine-tuning) significantly outperforms training directly on real-world data, especially for tail categories (screws).
  • The full pipeline achieved a screw IoU of 0.7842 on the real-world test data, compared to 0.4884 for a model trained only on real-world data without pre-training.
  • The patch-based attention module showed improvements in both simulation and real-world performance, achieving a screw IoU of 0.7968 in the real-world setting.
  • Qualitative results demonstrate that the fine-tuned model provides much better segmentation of screws compared to direct training on real-world data.

Limitations and Future Work

  • Focused Sampling Issue: The focused sampling strategy for imbalanced learning showed a decrease in performance on real-world data, likely due to the difference in data distribution between training and testing.
  • Weighting Loss Trade-offs: The loss weighting strategy improved synthetic data performance but slightly degraded real-world performance, suggesting a trade-off.
  • Future Directions: The authors plan to explore more robust backbones, investigate other attention-based learning methods for point clouds, and extend the approach to more complex disassembly tasks. They also aim to investigate more on the attention-based learning methods for point cloud data.

Summary updated

Abstract

On robotics computer vision tasks, generating and annotating large amounts of data from real-world for the use of deep learning-based approaches is often difficult or even impossible. A common strategy for solving this problem is to apply simulation-to-reality (sim2real) approaches with the help of simulated scenes. While the majority of current robotics vision sim2real work focuses on image data, we present an industrial application case that uses sim2real transfer learning for point cloud data. We provide insights on how to generate and process synthetic point cloud data in order to achieve better performance when the learned model is transferred to real-world data. The issue of imbalanced learning is investigated using multiple strategies. A novel patch-based attention network is proposed additionally to tackle this problem.
BibTeX
@inproceedings{wu2023sim,
  author = {Chengzhi Wu and Xuelei Bi and Julius Pfrommer and Alexander Cebulla and Simon Mangold and Jürgen Beyerer},
  title = {{Sim2real Transfer Learning for Point Cloud Segmentation: An Industrial Application Case on Autonomous Disassembly}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure