Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning

Wenshuai Zhao, Jorge Peña Queralta, Li Qingqing, Tomi WesterlundarXiv 2020 · arXiv

Abstract

Training robots in simulation offers speed and safety but faces the "Sim-to-Real gap," where policies fail on physical hardware due to unmodeled discrepancies. This challenge is magnified in collaborative multi-robot learning, where agents share experiences to update a common policy. This study investigates the impact of individual hardware variations—such as sensor noise and calibration errors—on collective learning performance. Using a simulated Kuka arm environment and Proximal Policy Optimization (PPO), we demonstrate that while systems are resilient to sensing errors, actuation inaccuracies (output errors) can cause learning to collapse, especially when perturbations are variable or widespread.

Introduction

The "Sim-to-Real gap" remains a significant hurdle in robotics. In collaborative settings, "identical" robots often possess unique hardware variations, such as misaligned sensors or mechanical wear. This research addresses whether these individual discrepancies "poison" the collective learning effort. We explore the robustness of shared policies when subsets of agents operate with "imperfect" hardware.

Methods and Experimental Setup

The study utilizes a synchronous multi-agent Deep Reinforcement Learning (DRL) framework based on the Proximal Policy Optimization (PPO) algorithm.

Collaborative learning architecture where multiple agents aggregate experiences to update a common model.

Simulation Environment

The experiments were conducted in a custom-modified Bullet physics environment featuring a Kuka robotic arm tasked with reaching specific coordinates. The setup involved 30 agents training in parallel, contributing to a single global model. The authors have open-sourced the environment as TIERS/NoisyKukaReacher.

Perturbation Models

To simulate real-world noise, four types of disturbances were introduced to varying subsets of agents:

  1. Fixed Input (IF): Constant sensing offsets (e.g., camera misalignment).
  2. Variable Input (IV): Random sensing noise (e.g., sensor jitter).
  3. Fixed Output (OF): Constant actuation offsets (e.g., joint calibration errors).
  4. Variable Output (OV): Random actuation noise (e.g., mechanical wear).

Results and Analysis

The experiments revealed critical thresholds for policy convergence under different noise regimes.

Learning curves across various perturbation scenarios (IF, IV, OF, OV) showing the impact of noise type and agent count on mean score.

Sensing vs. Actuation Robustness

A primary finding is that collaborative learning is significantly more sensitive to actuation (output) errors than sensing (input) errors. Agents successfully compensated for "lying" sensors, reaching near-perfect performance (mean score ~ -2) even with widespread fixed sensing errors (IF5, IF15, IF25). In contrast, actuation errors—particularly variable ones—frequently led to unstable behavior and failure to converge.

The Impact of Noise Variance

Fixed offsets were relatively easy for the neural network to "bake into" the policy. However, variable noise introduced high variance in experience updates, preventing the model from stabilizing. For instance, when 15 or more agents suffered from variable actuation errors (OV15, OV25), performance plummeted to baseline levels (mean score ~ -6).

The "Majority Rule" of Convergence

The system exhibited a threshold-based behavior. If only a small fraction (e.g., 5/30) of agents were faulty, the global model often converged by ignoring outlier data. However, when faulty agents became the majority, or when variable sensing noise affected over 50% of the group, the "noise" overwhelmed the "signal," leading to learning failure.

Conclusion and Future Work

This research highlights that actuation repeatability is more critical than sensor precision for stable collaborative learning. While simulation provides a controlled testbed, future work must validate these findings on physical hardware and explore more complex tasks like grasping or assembly. These results serve as a baseline for developing "perturbation-aware" RL algorithms capable of filtering data from faulty agents.

Summary updated

Abstract

Current research directions in deep reinforcement learning include bridging the simulation-reality gap, improving sample efficiency of experiences in distributed multi-agent reinforcement learning, together with the development of robust methods against adversarial agents in distributed learning, among many others. In this work, we are particularly interested in analyzing how multi-agent reinforcement learning can bridge the gap to reality in distributed multi-robot systems where the operation of the different robots is not necessarily homogeneous. These variations can happen due to sensing mismatches, inherent errors in terms of calibration of the mechanical joints, or simple differences in accuracy. While our results are simulation-based, we introduce the effect of sensing, calibration, and accuracy mismatches in distributed reinforcement learning with proximal policy optimization (PPO). We discuss on how both the different types of perturbances and how the number of agents experiencing those perturbances affect the collaborative learning effort. The simulations are carried out using a Kuka arm model in the Bullet physics engine. This is, to the best of our knowledge, the first work exploring the limitations of PPO in multi-robot systems when considering that different robots might be exposed to different environments where their sensors or actuators have induced errors. With the conclusions of this work, we set the initial point for future work on designing and developing methods to achieve robust reinforcement learning on the presence of real-world perturbances that might differ within a multi-robot system.
BibTeX
@misc{zhao2020towards,
  author = {Wenshuai Zhao and Jorge Peña Queralta and Li Qingqing and Tomi Westerlund},
  title = {{Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning}},
  year = {2020},
  eprint = {2008.07875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.07875},
}
Figure