This paper details the winning entry for Phase 1 of the 2021 Real Robot Challenge (RRC). The authors demonstrate that a "pure" reinforcement learning (RL) approach can outperform traditional, heavily engineered robotic control systems in complex, dexterous manipulation tasks.
Introduction
Dexterous manipulation—the ability of a robot to handle objects with finger-like precision—remains a significant challenge in robotics. Traditional methods usually rely on complex inverse kinematics and pre-defined "motion primitives." However, these methods often struggle in unstructured environments or when the task requires high degrees of freedom. The Real Robot Challenge (RRC) tasks participants with controlling a "TriFinger" robot to move a cube along a specific 3D path.
Methodology
The authors utilized Deep Deterministic Policy Gradients (DDPG) combined with Hindsight Experience Replay (HER), featuring several key modifications to overcome the "lifting" problem and the "sim-to-real" gap:
- Hybrid Reward System: The reward was split into a sparse reward for horizontal (x-y) movement and a dense reward for vertical (z) movement. This "nudged" the robot to keep the cube off the ground.
- Selective HER: HER was only applied to x-y coordinates. By not applying it to the vertical axis, the agent was prevented from being "satisfied" with dropping the cube, forcing it to learn to maintain height.
- Sim-to-Real Pipeline: Policies were trained entirely in simulation using Domain Randomization (DR). By varying physics parameters like friction and mass during training, the policy became robust enough to handle the imperfect physics of the real world.
Emergent Strategies and Results
One of the most compelling findings was that the RL agent "invented" distinct grasping styles without human intervention. The authors identified three primary strategies: Pushing, Cradling (using the robot's forearms), and Pinching (using fingertips).
Key Results
- First Place: The authors' team ('thriftysnipe') won Phase 1 of the 2021 RRC.
- Performance: Their "Pinching" policy achieved a score of -11,489 on the real robot, significantly outperforming the runner-up's score of -14,714.
- Efficiency: The modified algorithm converged in roughly 2/3 the time of standard HER.
Conclusion
While the approach was successful, limitations remain. Real-world performance was lower than in simulation, suggesting that fine-tuning on real-robot data is necessary. Additionally, the 10 million steps required for training highlight a need for improved sample efficiency. Future work will likely explore Model-Based Reinforcement Learning to reduce the data requirements for training directly on physical hardware.
Summary updated