Real Robot Challenge using Deep Reinforcement Learning.

Robert McCarthy, Francisco Roldán Sánchez, Kevin McGuinness, Noel E. O’Connor, Stephen J. Redmond

This paper details the winning entry for Phase 1 of the 2021 Real Robot Challenge (RRC). The authors demonstrate that a "pure" reinforcement learning (RL) approach can outperform traditional, heavily engineered robotic control systems in complex, dexterous manipulation tasks.

Introduction

Dexterous manipulation—the ability of a robot to handle objects with finger-like precision—remains a significant challenge in robotics. Traditional methods usually rely on complex inverse kinematics and pre-defined "motion primitives." However, these methods often struggle in unstructured environments or when the task requires high degrees of freedom. The Real Robot Challenge (RRC) tasks participants with controlling a "TriFinger" robot to move a cube along a specific 3D path.

The TriFinger robot platform used in the Real Robot Challenge.

Methodology

The authors utilized Deep Deterministic Policy Gradients (DDPG) combined with Hindsight Experience Replay (HER), featuring several key modifications to overcome the "lifting" problem and the "sim-to-real" gap:

  • Hybrid Reward System: The reward was split into a sparse reward for horizontal (x-y) movement and a dense reward for vertical (z) movement. This "nudged" the robot to keep the cube off the ground.
  • Selective HER: HER was only applied to x-y coordinates. By not applying it to the vertical axis, the agent was prevented from being "satisfied" with dropping the cube, forcing it to learn to maintain height.
  • Sim-to-Real Pipeline: Policies were trained entirely in simulation using Domain Randomization (DR). By varying physics parameters like friction and mass during training, the policy became robust enough to handle the imperfect physics of the real world.

Emergent Strategies and Results

One of the most compelling findings was that the RL agent "invented" distinct grasping styles without human intervention. The authors identified three primary strategies: Pushing, Cradling (using the robot's forearms), and Pinching (using fingertips).

Emergent manipulation strategies: (a) Pushing, (b) Cradling, and (c) Pinching.

Key Results

  • First Place: The authors' team ('thriftysnipe') won Phase 1 of the 2021 RRC.
  • Performance: Their "Pinching" policy achieved a score of -11,489 on the real robot, significantly outperforming the runner-up's score of -14,714.
  • Efficiency: The modified algorithm converged in roughly 2/3 the time of standard HER.

Conclusion

While the approach was successful, limitations remain. Real-world performance was lower than in simulation, suggesting that fine-tuning on real-robot data is necessary. Additionally, the 10 million steps required for training highlight a need for improved sample efficiency. Future work will likely explore Model-Based Reinforcement Learning to reduce the data requirements for training directly on physical hardware.

Summary updated

Abstract

This paper details our winning submission to Phase 1 of the 2021 Real Robot Challenge, a challenge in which a three fingered robot must carry a cube along specified goal trajectories. To solve Phase 1, we use a pure reinforcement learning approach which requires minimal expert knowledge of the robotic system or of robotic grasping in general. A sparse goal-based reward is employed in conjunction with Hindsight Experience Replay to teach the control policy to move the cube to the desired x and y coordinates. Simultaneously, a dense distance-based reward is employed to teach the policy to lift the cube to the desired z coordinate. The policy is trained in simulation with domain randomization before being transferred to the real robot for evaluation. Although performance tends to worsen after this transfer, our best trained policy can successfully lift the real cube along goal trajectories via the use of an effective pinching grasp. Our approach outperforms all other submissions, including those leveraging more traditional robotic control techniques, and is the first learning-based approach to solve this challenge.
BibTeX
@inproceedings{mccarthy2021real,
  author = {Robert McCarthy and Francisco Roldán Sánchez and Kevin McGuinness and Noel E. O’Connor and Stephen J. Redmond},
  title = {{Real Robot Challenge using Deep Reinforcement Learning.}},
  booktitle = {AICS},
  year = {2021},
  url = {https://openreview.net/forum?id=bGFZdnPMAm3},
}
Figure