Throwing Objects into A Moving Basket While Avoiding Obstacles

Hamidreza Kasaei, Hamidreza Kasaei 0001, Mohammadreza Kasaei

TL;DR

This paper presents a novel deep reinforcement learning (RL) approach that enables robots to precisely throw objects into moving baskets, even when obstacles are present in the environment. By formulating object throwing as an RL problem, the robot learns to dynamically adjust its throwing parameters, achieving impressive accuracy and generalization capabilities without explicit analytical modeling of object physics.

What Problem Does This Paper Address?

Humans are remarkably adept at throwing objects, a skill that allows us to quickly place items in desired locations, even those outside our immediate reach. While robots can perform pick-and-place tasks, replicating this dynamic throwing ability has been a significant challenge. Previous attempts often relied on analytical models or hand-coded behaviors, which struggle with the complexities of varying object shapes, masses, and especially dynamic environments with moving targets and obstacles. This paper tackles the critical gap of enabling robots to perform accurate, obstacle-aware object throwing into a moving receptacle, a task that requires sophisticated perception, prediction, and control.

Methods

The core of this work lies in framing the object throwing task as a Markov Decision Process (MDP) and solving it using an off-policy reinforcement learning algorithm. Specifically, the authors employed Deep Deterministic Policy Gradient (DDPG) and Soft Actor-Critic (SAC).

Here's a breakdown of their approach:

  • State Representation: The robot's state is a rich feature vector encompassing its proprioception (joint angles), the pose and distance to the obstacle and target basket, and timing parameters related to the throw (release time, trajectory duration).
  • Action Space: The actions are defined by parameters that directly modulate a kernel trajectory for throwing: initial and final shoulder joint values, trajectory execution duration, and object release time.
  • Perception: A crucial component is the robot's perception system, which uses an RGB-D camera to detect and track objects (target, basket, obstacles), estimate their poses and sizes, and predict stable grasp poses. This information forms the basis of the MDP's state.
  • RL Formulation: The RL agent learns to map states to actions (throwing parameters) to maximize a reward function. Success is rewarded, collisions with obstacles incur a severe penalty, and throws that miss the basket are penalized based on distance.
  • Simulation-to-Real Transfer: To address safety concerns and the difficulty of real-world training, the policy was initially trained in a Gazebo simulation environment that closely mirrors the real robot setup. The learned policy was then directly transferred to the real robot without fine-tuning.

Datasets

The paper does not explicitly mention pre-existing datasets for training. Instead, they generated their own training data through extensive simulations.

For evaluation, they used:

  • Simulated Objects: 10 objects with varying materials, shapes, and sizes. Five were used for training and five "unseen" objects for testing.
  • Real Objects: 5 household objects distinct in size and shape from the simulated training objects.

The experiments were conducted across three tasks with increasing difficulty:

  1. Obstacle-free throwing into a static basket.
  2. Obstacle-free throwing into a moving basket.
  3. Throwing into a moving basket while avoiding an obstacle.

Key Insights and Ideas

  • RL for Dynamic Throwing: The paper convincingly demonstrates that RL is a powerful paradigm for learning complex dynamic manipulation tasks like object throwing, especially when analytical modeling is difficult. The learned policy can adapt to variations in object properties and environmental conditions.
  • Effective Simulation-to-Real Transfer: The success of directly transferring a policy trained in simulation to a real robot without fine-tuning is a significant finding. This highlights the fidelity of their simulation environment and the robustness of the learned policy.
  • Generalization to Unseen Objects: The robot's ability to generalize its throwing skill to objects it has not encountered during training is a testament to the learned representations and the effectiveness of the RL approach.
  • Obstacle Avoidance as an Integrated Task: By incorporating obstacle detection and avoidance directly into the RL framework, the robot learns to navigate these challenges naturally as part of the throwing process, rather than as a separate post-processing step.

Key Results

The paper reports impressive throwing success rates across all tasks and conditions:

  • Task 1 (Static Basket): SAC achieved up to 94% accuracy for seen objects and 91% for unseen objects in simulation, and 90% in real-world tests.
  • Task 2 (Moving Basket): SAC achieved 91% accuracy for seen objects and 86% for unseen objects in simulation. In real-world tests, it maintained competitive performance.
  • Task 3 (Moving Basket with Obstacle): This most challenging task saw SAC achieve 86% accuracy for seen objects and 83% for unseen objects in simulation, and 80% in real-world tests.

Across all tasks, the SAC algorithm consistently outperformed DDPG and Behavior Cloning (BC), especially for unseen objects and in more complex scenarios. The learned policy demonstrated robustness to noise and uncertainty in the real-world.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Failure Cases: Failures were attributed to inaccurate parameter prediction, inaccurate object tracking (especially with fast-moving users), lag in gripper command execution, and unstable grasp poses for heterogeneous objects.
  • Future Work: The authors propose enhancing throwing performance by incorporating additional sensory modalities, such as force or tactile sensors. These could help the robot grasp objects more stably and further refine its throwing parameters. They also aim to investigate the possibility of further improving generalization and robustness.

Summary updated

Abstract

The capabilities of a robot will be increased significantly by exploiting throwing behavior. In particular, throwing will enable robots to rapidly place the object into the target basket, located outside its feasible kinematic space, without traveling to the desired location. In previous approaches, the robot often learned a parameterized throwing kernel through analytical approaches, imitation learning, or hand-coding. There are many situations in which such approaches do not work/generalize well due to various object shapes, heterogeneous mass distribution, and also obstacles that might be presented in the environment. It is obvious that a method is needed to modulate the throwing kernel through its meta parameters. In this paper, we tackle object throwing problem through a deep reinforcement learning approach that enables robots to precisely throw objects into moving baskets while there are obstacles obstructing the path. To the best of our knowledge, we are the first group that addresses throwing objects with obstacle avoidance. Such a throwing skill not only increases the physical reachability of a robot arm but also improves the execution time. In particular, the robot detects the pose of the target object, basket, and obstacle at each time step, predicts the proper grasp configuration for the target object, and then infers appropriate parameters to throw the object into the basket. Due to safety constraints, we develop a simulation environment in Gazebo to train the robot and then use the learned policy in real-robot directly. To assess the performers of the proposed approach, we perform extensive sets of experiments in both simulation and real robots in three scenarios. Experimental results showed that the robot could precisely throw a target object into the basket outside its kinematic range and generalize well to new locations and objects without colliding with obstacles.
BibTeX
@inproceedings{kasaei2023throwing,
  author = {Hamidreza Kasaei and Hamidreza Kasaei and Mohammadreza Kasaei},
  title = {{Throwing Objects into A Moving Basket While Avoiding Obstacles}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160215},
}
Figure