"What, not how": Solving an under-actuated insertion task from scratch

Giulia Vezzani, Michael Neunert, Markus Wulfmeier, Rae Jeong, Thomas Lampe, Noah Y. Siegel, Roland Hafner, Abbas Abdolmaleki, Martin A. Riedmiller, Francesco NoriarXiv 2020 · OpenReview

TL;DR

This paper tackles the challenge of teaching robots complex manipulation tasks by focusing on the "what" (the goal) rather than the "how" (the specific steps). They introduce a difficult peg-in-hole insertion task requiring multiple skills and demonstrate that a multi-task reinforcement learning approach can learn this task from scratch, even on a real robot with limited data, leading to emergent, intelligent behaviors.

What Problem Does This Paper Address?

Current reinforcement learning (RL) approaches in robotics often break down complex manipulation into simpler, single-skill tasks (e.g., just grasping or just inserting a pre-grasped object). This means the robot learns how to perform a specific action, but not necessarily what the overall goal is or how to combine multiple skills to achieve it. This paper addresses this gap by proposing a challenging, under-actuated peg-in-hole insertion task that necessitates the integration of several distinct manipulation skills: grasping, reorienting, and inserting. The key difficulty lies in the fact that the task only provides a reward for successful insertion (the "what"), with no intermediate rewards for achieving specific sub-goals like correct peg orientation. This forces the agent to learn the necessary pre-conditions and skills autonomously.

Methods

The authors tackle this problem using a multi-task reinforcement learning framework called Scheduled Auxiliary Control (SAC-X), augmented with Regularized Hierarchical Policy Optimization (RHPO).

  • SAC-X: This framework allows an agent to learn multiple tasks (or "intentions") simultaneously. It uses auxiliary tasks (like reaching, grasping, lifting) to guide exploration in the main, sparse-reward task (peg insertion). The agent learns to perform these auxiliary tasks, but the primary goal is still to solve the main task.
  • RHPO: This algorithm is designed for robust training of hierarchical policies in multi-task settings. It allows for shared low-level components across tasks, learning task-independent skills, while a high-level controller manages task-specific information.

Crucially, the authors do not provide any reward signal for peg reorientation, which is a critical pre-condition for successful insertion in their task. This forces the agent to discover the necessity and method of reorientation itself.

Datasets

The experiments were conducted in both simulation using the MuJoCo physics simulator and on a single real Rethink Robotics Sawyer robot arm. The paper does not introduce a new benchmark dataset but rather a novel, challenging task setup for evaluating RL algorithms.

Key Insights and Ideas

  • Focusing on "What" over "How" is Crucial for Complex Tasks: By only rewarding the final goal (successful insertion) and not specifying intermediate steps or correct orientations, the agent is forced to learn a more holistic and robust strategy, leading to emergent behaviors.
  • Under-actuation Drives Intelligent Behavior: The task is designed such that the robot's action space makes direct reorientation of the peg difficult. This under-actuation is key to observing intelligent, emergent strategies like dropping and poking to achieve the desired orientation.
  • Multi-task RL is Essential for Exploration in Sparse-Reward Environments: Standard RL struggles with exploration in tasks with infrequent rewards. SAC-X, by providing auxiliary tasks, significantly aids the agent in discovering useful behaviors.
  • Emergent Strategies are Surprising and Effective: The robot developed sophisticated, non-obvious methods for reorienting the peg (e.g., dropping it strategically) that were not explicitly programmed or rewarded, demonstrating the power of learning from scratch.

Key Results

  • The proposed multi-task RL approach (SAC-X + RHPO) successfully learned the complex peg-in-hole insertion task from scratch, both in simulation and on a single real robot.
  • On the real robot, the agent achieved a high success rate (up to 0.85) after approximately 21,000 episodes of training.
  • When trained with reloaded data from a previous run, the agent achieved even faster learning, with successful insertions occurring after only 500 episodes.
  • The agent developed emergent, under-actuated reorientation strategies, such as "reorientation by dropping" and "reorientation by poking," which were not explicitly designed into the task.

Limitations and Future Work

  • Data Efficiency: While the multi-task approach improves data efficiency compared to single-task RL, training on the real robot still requires a significant number of episodes (thousands). The authors suggest investigating Batch RL algorithms for further speedups.
  • Transferability of Skills: The paper notes that some learned skills, like "reorientation by poking," were abandoned during training. Future work could explore storing and transferring these skills to accelerate learning on new tasks.
  • Simulation Fidelity: The authors acknowledge that accurate simulation of complex physical interactions remains challenging, and the simulation task can be harder than the real-world counterpart due to unrealistic behaviors.
  • Generalizability: While the task is complex, it is still a specific peg-in-hole scenario. Applying the learned framework to a broader range of manipulation tasks is a natural next step.

Summary updated

Abstract

Robot manipulation requires a complex set of skills that need to be carefully combined and coordinated to solve a task. Yet, most ReinforcementLearning (RL) approaches in robotics study tasks which actually consist only of a single manipulation skill, such as grasping an object or inserting a pre-grasped object. As a result the skill ('how' to solve the task) but not the actual goal of a complete manipulation ('what' to solve) is specified. In contrast, we study a complex manipulation goal that requires an agent to learn and combine diverse manipulation skills. We propose a challenging, highly under-actuated peg-in-hole task with a free, rotational asymmetrical peg, requiring a broad range of manipulation skills. While correct peg (re-)orientation is a requirement for successful insertion, there is no reward associated with it. Hence an agent needs to understand this pre-condition and learn the skill to fulfil it. The final insertion reward is sparse, allowing freedom in the solution and leading to complex emerging behaviour not envisioned during the task design. We tackle the problem in a multi-task RL framework using Scheduled Auxiliary Control (SAC-X) combined with Regularized Hierarchical Policy Optimization (RHPO) which successfully solves the task in simulation and from scratch on a single robot where data is severely limited.
BibTeX
@misc{vezzani2020what,
  author = {Giulia Vezzani and Michael Neunert and Markus Wulfmeier and Rae Jeong and Thomas Lampe and Noah Y. Siegel and Roland Hafner and Abbas Abdolmaleki and Martin A. Riedmiller and Francesco Nori},
  title = {{"What, not how": Solving an under-actuated insertion task from scratch}},
  year = {2020},
  eprint = {2010.15492},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.15492},
}
Figure