Reinforcement Learning to improve delta robot throws for sorting scrap metal

Arthur Louette, Gaspard Lambrechts, Damien Ernst, Eric Pirard, Godefroid DislairearXiv 2024 · DOI · arXiv

TL;DR

This paper introduces a novel reinforcement learning (RL) approach for improving the efficiency of scrap metal sorting using delta robots. By training robots to "throw" objects into bins rather than precisely placing them, they achieve a significant speedup and comparable accuracy to traditional methods, demonstrating the potential of RL for industrial automation.

What Problem Does This Paper Address?

The recycling and remanufacturing of end-of-life products require meticulous disassembly and sorting of components. While optical sorting exists, there's a growing need for higher throughput. Delta robots are being employed for this, but the traditional Pick-and-Place (PaP) method, while precise, has limitations in speed. This paper tackles the challenge of increasing sorting speed by exploring a Pick-and-Throw (PaT) strategy. Unlike PaP, where the target is a precise bin location, PaT involves throwing an object, which introduces complexities in controlling the release position and speed to ensure accuracy and minimize time. Existing analytical methods for PaT struggle with the inherent uncertainties and variability of scrap metal (irregular shapes, varying properties) and the limitations of parallel grippers, which are less suited than vacuum grippers for such tasks.

Methods

The core of this work is the application of reinforcement learning (RL) to learn a Pick-and-Throw (PaT) strategy. Specifically, they frame the problem as a contextual bandit problem, where the robot learns to make a single decision (release position and velocity) for each throw based on the current context (object and bin positions).

Key methodological contributions include:

  • Novel Simulation Environment: A new simulation environment is developed specifically for training RL agents on PaT strategies for parallel grippers. This environment incorporates domain randomization to improve sim-to-real transfer.
  • RL Algorithms: Three model-free RL algorithms are employed: TD3, SAC, and PPO. These algorithms learn directly from interactions within the simulation, bypassing the need for explicit dynamic modeling of the robot and environment.
  • Pick-and-Throw (PaT) Strategy: Instead of moving to a precise bin location, the robot learns to open its gripper at a specific position and with a specific velocity to throw the object towards the target bin. This is a departure from traditional PaP.
  • Reward Function Design: A novel reward function is designed to balance accuracy and speed. It rewards successful throws and penalizes time, with an additional incentive for successful throws of objects further from the bin, indirectly encouraging faster, more efficient throws. A baseline estimator for the PaP process time is also introduced to provide a comparative reward signal.
  • Domain Randomization: To enhance the transferability of learned policies to the real world, domain randomization is applied. This involves randomizing parameters such as bin positions, gripper opening delays, and object properties (mass, size, center of mass) during training.

Datasets

The paper does not explicitly mention or use pre-existing datasets for training. Instead, it relies on a newly developed simulation environment for generating training data. The experiments are conducted within this simulated environment, and the learned policies are then evaluated in both simulation and a real-world setting.

Key Insights and Ideas

  • RL for Dynamic Throwing: The study successfully demonstrates that RL can effectively learn complex dynamic tasks like throwing irregular objects, overcoming the limitations of analytical models that struggle with uncertainty.
  • PaT Outperforms PaP in Speed: The RL-based PaT strategy significantly speeds up the sorting process compared to the conventional Pick-and-Place (PaP) method, achieving a 51% throughput increase in simulation.
  • Sim-to-Real Transfer is Achievable: Through domain randomization, the policies trained in simulation can be transferred to a real-world delta robot with minimal performance degradation, highlighting the effectiveness of this technique.
  • Parallel Grippers are Viable for PaT: The work shows that PaT is feasible even with parallel grippers, which are typically less suited for handling varied scrap metal compared to vacuum grippers.
  • RL Adapts to Uncertainty: The RL approach inherently handles the variability in object properties and environmental conditions, which is a major advantage over model-based methods.

Key Results

  • Simulation Performance: The best RL agents achieved an accuracy of 89.32% and a throughput increase of 51% in simulation.
  • Real-World Performance: In real-world experiments, the RL-based PaT strategy achieved a success rate of 82.666% (compared to 100% for PaP) but with an average processing time of 332 ms, which is 48 ms faster than the PaP procedure, representing a 13% edge in cycle time.
  • Comparison to Existing PaT: The proposed RL-based PaT significantly outperformed the time-optimal s-curve PaT strategy proposed by Hassan et al. [9], which struggled to achieve success with parallel grippers and irregular objects.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Bin Position Sensitivity: The current approach is sensitive to the encoded position of the bin center, and the bin architecture in the benchmark was not optimized for throwing.
  • Ignoring Bin Shape: The current formulation does not consider the advantageous shape of some bins.
  • Independence of Throws: The problem is currently modeled as a Markov decision process where successive throws are independent. Future work could model the dependence between throws to optimize cumulative return and potentially improve time gains by adjusting trajectories.
  • Complex Movements: Delta robots are capable of more complex movements than simple linear ones. Future research could explore actions that parametrize more complex movements for PaT.
  • Real-World Fine-Tuning: While sim-to-real transfer was successful without fine-tuning, using real-world data for fine-tuning could further improve performance.

Summary updated

Abstract

This study proposes a novel approach based on reinforcement learning (RL) to enhance the sorting efficiency of scrap metal using delta robots and a Pick-and-Place (PaP) process, widely used in the industry. We use three classical model-free RL algorithms (TD3, SAC and PPO) to reduce the time to sort metal scraps. We learn the release position and speed needed to throw an object in a bin instead of moving to the exact bin location, as with the classical PaP technique. Our contribution is threefold. First, we provide a new simulation environment for learning RL-based Pick-and-Throw (PaT) strategies for parallel grippers. Second, we use RL algorithms for learning this task in this environment resulting in 89% accuracy while speeding up the throughput by 51% in simulation. Third, we evaluate the performances of RL algorithms and compare them to a PaP and a state-of-the-art PaT method both in simulation and reality, learning only from simulation with domain randomisation and without fine tuning in reality to transfer our policies. This work shows the benefits of RL-based PaT compared to PaP or classical optimization PaT techniques used in the industry.
BibTeX
@misc{louette2024reinforcement,
  author = {Arthur Louette and Gaspard Lambrechts and Damien Ernst and Eric Pirard and Godefroid Dislaire},
  title = {{Reinforcement Learning to improve delta robot throws for sorting scrap metal}},
  year = {2024},
  eprint = {2406.13453},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.13453},
}
Figure