RESPRECT: Speeding-up Multi-Fingered Grasping With Residual Reinforcement Learning

Federico Ceola, Lorenzo Rosasco, Lorenzo Natale

TL;DR

RESPRECT (RESidual learning with PREtrained CriTics) significantly accelerates multi-fingered robotic grasping by learning a "residual" policy on top of a pre-trained general model. By reusing the "Critic" weights from the base model to jumpstart learning, the authors achieved a 5x speedup over training from scratch, enabling a real iCub humanoid robot to master grasping novel objects in as little as 30 minutes without any human demonstrations.

What Problem Does This Paper Address?

Teaching a robot with a human-like, multi-fingered hand to grasp objects is a "high-dimensional" nightmare. Unlike simple parallel-jaw grippers, dexterous hands have many degrees of freedom (DoFs) that must be coordinated perfectly.

While Deep Reinforcement Learning (DRL) can solve this in simulation, it usually requires millions of trials—far too many for a physical robot to endure without breaking. Existing shortcuts have major flaws:

  1. Sim-to-Real Transfer: Often fails because simulations don't perfectly mimic real-world physics (the "sim-to-real gap").
  2. Residual Reinforcement Learning (RRL): Usually requires a hand-tuned, classical controller as a "base" to build upon, but designing such a controller for a 9-DoF hand is incredibly difficult.
  3. Imitation Learning: Requires a human to provide hundreds of successful demonstrations, which is tedious and time-consuming.

Methods

The authors introduce RESPRECT, a novel RRL framework. Instead of building on a hand-coded controller, it builds on a pre-trained DRL policy that was trained on a wide variety of generic objects in simulation.

Key components include:

  • Additive Policy: The final action sent to the robot is the sum of the pre-trained policy's output and the new residual policy's output ($a_t = a_{PRE,t} + a_{RL,t}$). The pre-trained weights remain frozen, while the residual policy learns the specific "tweaks" needed for a new, unseen object.
  • Pre-trained Critics: This is the "secret sauce." In the Soft Actor-Critic (SAC) algorithm, "Critics" estimate how good an action is. RESPRECT initializes the residual policy's Critics using the weights of the pre-trained model. This "warm start" provides an immediate, intelligent signal for learning, rather than starting from random noise.
  • Visual Backbone: They utilize a Masked Autoencoder (MAE) with a ViT-Large architecture to extract visual features from the robot's head camera, which proved more effective for generalization than other popular models like CLIP.

Datasets

  • MSO (Scanned Objects Dataset): Used to train the "base" policy in simulation. It contains over 300 diverse household objects.
  • YCB-Video Dataset: A standard robotics benchmark. Seven specific objects (e.g., mustard bottle, cracker box, power drill) were used for evaluation in simulation.
  • Real-world Evaluation: The system was tested on the physical iCub robot using the mustard bottle and bleach cleanser from the YCB set.

Key Insights and Ideas

  • DRL on DRL: This is the first RRL approach to successfully use a DRL-trained policy as the base rather than a classical controller. This suggests that "imperfect" neural network policies are stable enough to serve as foundations for further learning.
  • The Power of the Critic: The most significant speedup didn't come from the residual architecture alone, but from reusing the Critic weights. By starting with a "Critic" that already understands the basics of what a good grasp looks like, the agent avoids the aimless exploration phase that plagues most RL.
  • No Demos Required: Unlike previous state-of-the-art methods (like G-PAYN), RESPRECT reaches high success rates without needing any human-provided grasping demonstrations to seed the memory buffer.

Key Results

  • 5x Speedup: In simulation, RESPRECT reached peak performance in 1 million timesteps, compared to the 5 million required by the previous state-of-the-art (G-PAYN).
  • Real-World Efficiency: The iCub robot learned to grasp a novel mustard bottle in ~2.5 hours (~10,000 steps) and a bleach cleanser in just ~30 minutes (~1,500 steps).
  • Superior Adaptation: RESPRECT consistently outperformed standard fine-tuning and Meta-Reinforcement Learning (MetaRL) baselines, which often struggled to adapt to the complex contact physics of multi-fingered hands.

Limitations and Future Work

  • Reactive Recovery: The current policy is somewhat "open-loop" during the final stage of the grasp. If the object slips or the initial approach fails, the robot struggles to adjust mid-motion.
  • Static Perception: The system assumes the object's position is relatively static once the episode begins.
  • Future Directions: The authors plan to integrate a class-agnostic object tracker to allow for more reactive, "closed-loop" grasping that can handle moving objects or mid-grasp failures. They also aim to refine the reward function to further reduce training time on the physical robot.

Summary updated

Abstract

Deep Reinforcement Learning (DRL) has proven effective in learning control policies using robotic grippers, but much less practical for solving the problem of grasping with dexterous hands – especially on real robotic platforms – due to the high dimensionality of the problem. In this letter, we focus on the multi-fingered grasping task with the anthropomorphic hand of the iCub humanoid. We propose the RESidual learning with PREtrained CriTics (RESPRECT) method that, starting from a policy pre-trained on a large set of objects, can learn a residual policy to grasp a novel object in a fraction ($\sim 5 \times$ faster) of the timesteps required to train a policy from scratch, without requiring any task demonstration. To our knowledge, this is the first Residual Reinforcement Learning (RRL) approach that learns a residual policy on top of another policy pre-trained with DRL. We exploit some components of the pre-trained policy during residual learning that further speed-up the training. We benchmark our results in the iCub simulated environment, and we show that RESPRECT can be effectively used to learn a multi-fingered grasping policy on the real iCub robot.
BibTeX
@article{ceola2024resprect,
  author = {Federico Ceola and Lorenzo Rosasco and Lorenzo Natale},
  title = {{RESPRECT: Speeding-up Multi-Fingered Grasping With Residual Reinforcement Learning}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2024},
  doi = {10.1109/lra.2024.3363532},
}

Loading related content...

Figure