A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning

Mel Vecerík, Oleg Sushkov, David Barker, Thomas Rothörl, Todd Hester, Jonathan ScholzICRA 2019 · DOI

TL;DR

This paper presents EDRIAD, a deep reinforcement learning approach that enables robots to perform insertion tasks with variable socket positions using only raw visual and haptic data. By combining a standard DDPG algorithm with minor modifications like behavior cloning and novel critic losses, EDRIAD achieves robust and efficient insertion learning on real robots in under 20 minutes, significantly outperforming prior methods.

What Problem Does This Paper Address?

Object insertion is a fundamental yet challenging task in robotics, crucial for manufacturing and assembly. Traditional methods often rely on precise knowledge of the socket's position and geometry, requiring complex, hand-engineered solutions tailored to each specific problem. This makes them brittle to variations in socket position or object dynamics, and difficult for non-experts to configure. While deep reinforcement learning (RL) offers a promising avenue for learning such complex manipulation skills, existing RL approaches have been too sample-inefficient for real-world robots or require high-level object features (like known socket poses), limiting their practical applicability. This paper aims to bridge this gap by developing an RL method that can learn insertion tasks efficiently and robustly from raw sensory inputs, even with significant variability in the target's position.

Methods

The core of EDRIAD is built upon the Deep Deterministic Policy Gradient (DDPG) algorithm, an off-policy actor-critic method. The authors introduce several key modifications to enhance its performance and suitability for real-world insertion tasks:

  • Visual Features and Rewards: Instead of relying on explicit object tracking, EDRIAD uses a convolutional neural network (CNN) trained to classify successful insertions from in-hand camera images. The internal features of this CNN are then used as visual inputs for the robot's policy. This approach avoids the need for complex vision systems.
  • Behavior Cloning Loss: A supervised behavior cloning loss is added to the actor network. This encourages the policy to mimic actions from expert demonstrations, providing a strong initial signal and improving learning efficiency, especially in the early stages.
  • Demonstration-Critic Losses: Two novel losses are introduced for the critic network:
  • Demonstration-Classification Loss: This loss trains the critic to distinguish between positive (successful) and negative (failed) demonstrations, helping it extract more information from the provided data.
  • Episode-Progress Loss: This loss trains the critic to estimate the progress within an episode, providing a more nuanced understanding of the task's progression.
  • Distributional Q-function: The standard temporal-difference (TD) loss for the critic is replaced with a distributional loss. This approach models the distribution of Q-values, leading to more stable and robust learning.
  • Pretraining: The agent is pretrained using the collected demonstration data before interacting with the environment. This significantly speeds up the learning process.

Datasets

The paper does not explicitly mention pre-existing public datasets for training or evaluation. Instead, they collected their own demonstration data for two specific insertion tasks:

  • Peg Insertion Task: A narrow-clearance peg-in-hole task (0.5 mm clearance), inspired by the Siemens challenge.
  • Clip Insertion Task: A deformable clip insertion task into square holes.

For both tasks, they collected positive demonstrations (successful insertions) and negative demonstrations (failed attempts or probing of failure modes). The socket position was randomized during episodes to simulate real-world variability.

Key Insights and Ideas

  • Vision is Crucial for Variability: The experiments strongly suggest that visual features are essential for robots to reliably perform insertion tasks when the socket's position is variable. Without vision, robots resort to less efficient "probing" behaviors.
  • Minor RL Tweaks Yield Major Gains: The paper demonstrates that relatively minor modifications to a standard DDPG algorithm, particularly the inclusion of behavior cloning and specialized critic losses, can dramatically improve sample efficiency and robustness for complex manipulation tasks.
  • Demonstrations as a Powerful Bootstrap: Leveraging human demonstrations, both for pretraining and for supervised losses, is a highly effective strategy for overcoming the exploration challenges inherent in sparse-reward RL problems.
  • End-to-End Learning from Raw Pixels is Feasible: The success of using internal CNN features directly for control, without explicit object tracking or feature engineering, highlights the potential of end-to-end deep learning for robotics.

Key Results

  • Rapid Learning on Real Robots: EDRIAD can learn to solve both the peg and clip insertion tasks on a real robot in under 20 minutes of interaction time.
  • High Success Rates: The approach achieves high success rates: 97% for the peg insertion task and 77% for the clip insertion task on the randomized versions.
  • Robustness to Variability: EDRIAD demonstrates robustness to variations in socket position and orientation, a key challenge for prior methods.
  • Efficiency Gains: The proposed approach reduces training time by approximately 60x compared to a previous DDPG with demonstrations (DDPGfD) approach.
  • Outperforming Behavior Cloning: EDRIAD significantly outperforms pure behavior cloning, learning to insert objects faster and more reliably than the supervised policy.

Limitations and Future Work

  • Compliance Controller Still Required: While EDRIAD learns the high-level policy, it still relies on a "carefully-engineered compliance controller" for low-level joint velocity control.
  • Hyperparameter Tuning: The approach still requires considerable hyperparameter tuning.
  • Grasping Assumption: The method assumes the object (plug) is already firmly grasped by the robot prior to the insertion phase.
  • Future Work: The authors suggest exploring simpler loss functions, methods to reduce hyperparameter sensitivity, and more principled ways to extract visual features. They also aim to address the need for a pre-defined compliance controller and the assumption of a pre-grasped object.

Summary updated

BibTeX
@inproceedings{vecerik2019practical,
  author = {Mel Vecerík and Oleg Sushkov and David Barker and Thomas Rothörl and Todd Hester and Jonathan Scholz},
  title = {{A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  doi = {10.1109/icra.2019.8794074},
}
Figure