Abstract
Deep Reinforcement Learning (DRL) has proven effective in learning control policies using robotic grippers, but much less practical for solving the problem of grasping with dexterous hands -- especially on real robotic platforms -- due to the high dimensionality of the problem. In this work, we focus on the multi-fingered grasping task with the anthropomorphic hand of the iCub humanoid. We propose the RESidual learning with PREtrained CriTics (RESPRECT) method that, starting from a policy pre-trained on a large set of objects, can learn a residual policy to grasp a novel object in a fraction ($\sim 5 \times$ faster) of the timesteps required to train a policy from scratch, without requiring any task demonstration. To our knowledge, this is the first Residual Reinforcement Learning (RRL) approach that learns a residual policy on top of another policy pre-trained with DRL. We exploit some components of the pre-trained policy during residual learning that further speed-up the training. We benchmark our results in the iCub simulated environment, and we show that RESPRECT can be effectively used to learn a multi-fingered grasping policy on the real iCub robot.
Introduction
Teaching a robot with a human-like, multi-fingered hand to grasp objects is a "high-dimensional" challenge. Unlike simple parallel-jaw grippers, dexterous hands have many degrees of freedom (DoFs) that must be coordinated perfectly. While Deep Reinforcement Learning (DRL) can solve this in simulation, it usually requires millions of trials—far too many for a physical robot to endure without mechanical failure.
Existing approaches to mitigate this complexity have significant drawbacks. Sim-to-real transfer often fails due to the "sim-to-real gap" in physics modeling. Traditional Residual Reinforcement Learning (RRL) typically relies on hand-tuned, classical controllers as a base, which are difficult to design for 9-DoF hands. Finally, imitation learning requires extensive human demonstrations, which are costly and time-consuming to collect.
The RESPRECT Framework
The authors introduce RESPRECT (RESidual learning with PREtrained CriTics), a novel RRL framework that builds upon a pre-trained DRL policy rather than a hand-coded controller. This base policy is trained on a wide variety of generic objects in simulation to provide a foundational grasping capability.
The framework relies on three key technical innovations:
- Additive Policy: The final control action is the sum of the frozen pre-trained policy's output and the learnable residual policy's output ($a_t = a_{PRE,t} + a_{RL,t}$). This allows the system to learn specific "tweaks" for novel objects while retaining general knowledge.
- Pre-trained Critics: In the Soft Actor-Critic (SAC) algorithm, "Critics" estimate the value of actions. RESPRECT initializes the residual policy's Critics using the weights of the pre-trained model. This "warm start" provides an immediate, intelligent signal for learning, bypassing the aimless exploration phase typical of RL.
- Visual Backbone: The system utilizes a Masked Autoencoder (MAE) with a ViT-Large architecture to extract visual features from the robot's head camera, which enhances generalization compared to models like CLIP.
Experimental Evaluation
The method was evaluated using the MSO (Scanned Objects Dataset) for base training and the YCB-Video Dataset for benchmarking. Seven specific objects, including a mustard bottle and a power drill, were used for simulated evaluation. Real-world validation was conducted on the physical iCub humanoid robot using objects like the bleach cleanser and mustard bottle.
Key Results and Insights
RESPRECT demonstrates that "imperfect" neural network policies are stable enough to serve as foundations for residual learning. A critical finding is that the most significant speedup stems from reusing the Critic weights, which guides exploration effectively from the start.
Key performance metrics include:
- 5x Speedup: In simulation, RESPRECT reached peak performance in approximately 1 million timesteps, whereas the previous state-of-the-art (G-PAYN) required 5 million.
- Real-World Efficiency: The iCub robot successfully learned to grasp a novel mustard bottle in ~2.5 hours and a bleach cleanser in just 30 minutes.
- No Demonstrations Required: Unlike G-PAYN, RESPRECT achieves high success rates without needing human-provided grasping demonstrations to seed the memory buffer.
Limitations and Future Work
Currently, the policy is somewhat "open-loop" during the final grasping stage, making it difficult for the robot to recover if an object slips. Additionally, the system assumes a static object position during the episode. Future work will focus on integrating class-agnostic object trackers to enable more reactive, "closed-loop" grasping and refining reward functions to further decrease real-world training time.
Summary updated