Abstract
Training robots in simulation offers speed and safety but faces the "Sim-to-Real gap," where policies fail on physical hardware due to unmodeled discrepancies. This challenge is magnified in collaborative multi-robot learning, where agents share experiences to update a common policy. This study investigates the impact of individual hardware variations—such as sensor noise and calibration errors—on collective learning performance. Using a simulated Kuka arm environment and Proximal Policy Optimization (PPO), we demonstrate that while systems are resilient to sensing errors, actuation inaccuracies (output errors) can cause learning to collapse, especially when perturbations are variable or widespread.
Introduction
The "Sim-to-Real gap" remains a significant hurdle in robotics. In collaborative settings, "identical" robots often possess unique hardware variations, such as misaligned sensors or mechanical wear. This research addresses whether these individual discrepancies "poison" the collective learning effort. We explore the robustness of shared policies when subsets of agents operate with "imperfect" hardware.
Methods and Experimental Setup
The study utilizes a synchronous multi-agent Deep Reinforcement Learning (DRL) framework based on the Proximal Policy Optimization (PPO) algorithm.
Simulation Environment
The experiments were conducted in a custom-modified Bullet physics environment featuring a Kuka robotic arm tasked with reaching specific coordinates. The setup involved 30 agents training in parallel, contributing to a single global model. The authors have open-sourced the environment as TIERS/NoisyKukaReacher.
Perturbation Models
To simulate real-world noise, four types of disturbances were introduced to varying subsets of agents:
- Fixed Input (IF): Constant sensing offsets (e.g., camera misalignment).
- Variable Input (IV): Random sensing noise (e.g., sensor jitter).
- Fixed Output (OF): Constant actuation offsets (e.g., joint calibration errors).
- Variable Output (OV): Random actuation noise (e.g., mechanical wear).
Results and Analysis
The experiments revealed critical thresholds for policy convergence under different noise regimes.
Sensing vs. Actuation Robustness
A primary finding is that collaborative learning is significantly more sensitive to actuation (output) errors than sensing (input) errors. Agents successfully compensated for "lying" sensors, reaching near-perfect performance (mean score ~ -2) even with widespread fixed sensing errors (IF5, IF15, IF25). In contrast, actuation errors—particularly variable ones—frequently led to unstable behavior and failure to converge.
The Impact of Noise Variance
Fixed offsets were relatively easy for the neural network to "bake into" the policy. However, variable noise introduced high variance in experience updates, preventing the model from stabilizing. For instance, when 15 or more agents suffered from variable actuation errors (OV15, OV25), performance plummeted to baseline levels (mean score ~ -6).
The "Majority Rule" of Convergence
The system exhibited a threshold-based behavior. If only a small fraction (e.g., 5/30) of agents were faulty, the global model often converged by ignoring outlier data. However, when faulty agents became the majority, or when variable sensing noise affected over 50% of the group, the "noise" overwhelmed the "signal," leading to learning failure.
Conclusion and Future Work
This research highlights that actuation repeatability is more critical than sensor precision for stable collaborative learning. While simulation provides a controlled testbed, future work must validate these findings on physical hardware and explore more complex tasks like grasping or assembly. These results serve as a baseline for developing "perturbation-aware" RL algorithms capable of filtering data from faulty agents.
Summary updated