Sim-and-Real Reinforcement Learning for Manipulation: A Consensus-based Approach

Wenxing Liu, Hanlin Niu, Wei Pan, Guido Herrmann, Joaquín García CarrascoarXiv 2023 · arXiv

Abstract: Sim-and-real training is a promising alternative to sim-to-real training for robot manipulations. However, the current sim-and-real training is neither efficient, i.e., slow convergence to the optimal policy, nor effective, i.e., sizeable real-world robot data. Given limited time and hardware budgets, the performance of sim-and-real training is not satisfactory. In this paper, we propose a Consensus-based Sim-And-Real deep reinforcement learning algorithm (CSAR) for manipulator pick-and-place tasks, which shows comparable performance in both sim-and-real worlds. In this algorithm, we train the agents in simulators and the real world to get the optimal policies for both sim-and-real worlds. We found two interesting phenomenons: (1) Best policy in simulation is not the best for sim-and-real training. (2) The more simulation agents, the better sim-and-real training.

Introduction

Deep Reinforcement Learning (DRL) has shown great potential in robotics, but its deployment is often hindered by the "reality gap." While simulators offer a safe and inexpensive training ground, policies learned in simulation frequently fail when transferred to physical hardware. The traditional "Sim-to-Real" approach attempts to bridge this gap via sequential fine-tuning on real robots, but this process remains data-intensive and slow.

In this paper, we propose a shift from sequential transfer to simultaneous learning. Our Consensus-based Sim-And-Real (CSAR) algorithm trains simulated and real-world agents in parallel, using a consensus protocol to synchronize their learning. This approach achieves high success rates in pick-and-place tasks with significantly less real-world data than conventional methods.

The CSAR Framework

The core innovation of CSAR is treating the training process as a multi-agent consensus problem. Rather than training a single agent, the system employs $M$ agents—typically several simulated robots and one physical robot—that learn concurrently.

The CSAR framework enables simultaneous training of simulated and real-world agents through a consensus protocol.

Consensus Protocol

Each agent maintains its own neural network parameters. After each training iteration, the agents communicate using a Laplacian-based consensus protocol. This mechanism mathematically encourages the parameters of all agents to converge toward a shared, weighted average. In practice, this allows the real-world agent to leverage the high-throughput data generated in simulation, while the simulated agents are "anchored" by the physical constraints and feedback of the real world.

Suction Net Architecture

To perform pick-and-place tasks, we developed a lightweight "Suction Net" based on a ResNet-18 backbone. The network processes raw 3D visual data in the form of RGB-D heightmaps. It outputs a pixel-wise map of Q-values, where each pixel represents a potential suction point. The robot executes the action corresponding to the pixel with the highest predicted value.

Detailed architecture of the CSAR system, featuring the Suction Net and the consensus-based training loop.

Experimental Setup

Our experiments utilized a UR5 robot arm equipped with a Robotiq suction gripper.

  • Simulation: We used CoppeliaSim with the Bullet Physics engine. The environment featured 5cm cubes.
  • Real World: The physical setup included an Azure Kinect camera for vision and 6.5cm cubes for manipulation.
  • Generalization: To test the robustness of the learned policy, we introduced novel objects during evaluation, including cylinders and various household items not present during training.

Key Insights

Our investigation into sim-and-real training revealed several counter-intuitive findings:

  1. The Advantage of "Mediocre" Policies: We found that starting with a perfectly optimized simulation policy is often detrimental. A "mediocre" policy (approx. 50% success in sim) provides the necessary plasticity for the consensus mechanism to adapt to real-world physics. An over-optimized simulation policy can be too rigid to bridge the reality gap.
  2. Simulation as Indirect Noise: In standard RL, real robots often get trapped in local minima. In CSAR, the simulated agents provide a form of "indirect noise" through the consensus updates, effectively pulling the real agent out of repetitive failure loops.
  3. Scaling Benefits: Increasing the number of simulated agents improves the stability and speed of real-world training. More simulation "votes" help the system converge more reliably on a robust policy.

Results and Discussion

The CSAR algorithm demonstrated superior efficiency compared to traditional baselines. Using a configuration of three simulated agents and one real agent, the system achieved an 80% suction success rate in just 140 training steps. In contrast, a 1-sim/1-real setup required 260 steps to reach the same performance level.

Furthermore, the policy showed strong generalization capabilities. Despite being trained only on cubes, the robot successfully manipulated irregularly shaped objects with an 80% success rate. While CSAR is highly effective for discrete action spaces like suction-point selection, future work will explore its application to more complex, continuous manipulation tasks.

The experimental video is available at: https://youtu.be/mcHJtNIsTEQ

Summary updated

Abstract

Sim-and-real training is a promising alternative to sim-to-real training for robot manipulations. However, the current sim-and-real training is neither efficient, i.e., slow con-vergence to the optimal policy, nor effective, i.e., sizeable real-world robot data. Given limited time and hardware budgets, the performance of sim-and-real training is not satisfactory. In this paper, we propose a Consensus-based Sim-And-Real deep reinforcement learning algorithm (CSAR) for manipulator pick-and-place tasks, which shows comparable performance in both sim-and- real worlds. In this algorithm, we train the agents in simulators and the real world to get the optimal policies for both sim-and-real worlds. We found two interesting phenomenons: (1) Best policy in simulation is not the best for sim-and-real training. (2) The more simulation agents, the better sim-and-real training. The experimental video is available at: https://youtu.be/mcHJtNIsTEQ.
BibTeX
@misc{liu2023sim,
  author = {Wenxing Liu and Hanlin Niu and Wei Pan and Guido Herrmann and Joaquín García Carrasco},
  title = {{Sim-and-Real Reinforcement Learning for Manipulation: A Consensus-based Approach}},
  year = {2023},
  eprint = {2302.13423},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.13423},
}
Figure