Randomizing When Tasks Finish, Not How Motors Behave: Zero-Shot Transfer for Asynchronous Robot Teams

When a team of robots coordinates through temporally extended actions such as "drive to the shelf" or "hand over the tool", the team-level policy never sees motor torques or friction coefficients. It sees only when each teammate's current task ends, whether it succeeded, and in what order those completion events arrived. This proposal argues that, for such teams, the gap between simulation and reality that matters for coordination is almost entirely a gap in task-completion timing. Real robots are slower or faster than simulated ones, and they are persistently so, since one robot has a worn gearbox and another has a laggy radio. Their tasks sometimes fail. Near-simultaneous completions arrive in a different order than the simulator predicted. We propose to train team policies with randomization applied directly to this timing process: each robot gets a persistent, robot-specific slowdown, per-execution jitter, a failure probability, and a chance that close events swap order. The alternative, randomizing every robot's physics and letting timing emerge, is far more expensive.

The idea is cheap to implement. It sidesteps the blow-up in the number of randomized parameters when every robot in a fleet has its own physics. It also yields a testable claim: a timing-randomized policy should transfer to a physics-rich target, and to real robots, about as well as a policy trained with full per-robot physics randomization, at a fraction of the simulation cost. It should also transfer much better than policies trained with the uniform action-delay jitter used in current asynchronous multi-robot work. If the claim holds, it tells practitioners where to spend their randomization budget. If it fails, the failure identifies which physical effects leak through the task abstraction.

Background and the Gap

Asynchronous multi-agent reinforcement learning drops the shared clock of standard multi-agent learning. Decisions happen only when an individual agent finishes a macro-action, a scripted, temporally extended behavior with a random duration. Several methods make learning work in this event-driven setting. Macro-Action-Based Deep Multi-Agent Reinforcement Learning compresses trajectories to decision points and conditions value targets on teammates who are still busy. Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning introduces Mac-IAICC, which gives each agent its own centralized critic that updates only at that agent's action boundaries. Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning introduces ACAC. It replaces trajectory padding with per-agent history encoders stamped with physical time, which will turn out to matter here. ToMacVF factorizes values over agent progress. We reuse these methods as the learners. None of them randomizes anything during training. Mac-IAICC did reach physical Fetch and Turtlebot robots, but transfer rested entirely on the abstraction of hand-scripted macro-actions, and the file reports no quantitative real-world number for it.

Domain randomization means perturbing simulator parameters across training episodes so the policy must work across a whole family of worlds. It has been developed almost entirely for single robots. Sim-to-Real: Learning Agile Locomotion For Quadruped Robots and OpenAI et al. randomize dynamics for one body. SPiDR adds a pessimism penalty from the disagreement among a set of randomized dynamics models, again for a single robot. The closest single-robot precedent for our argument is Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization. That work shows that the dominant reality gap for a quadruped was timing: sensor streams arrive at different latencies. Randomizing those latencies nearly doubled real-world travel distance, from 444.7 cm to 859.9 cm. We take the same lesson to the team level, where timing concerns events between robots rather than sensors inside one robot.

Two papers provide the motivation and one constrains the design. Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning shows that shared learning tolerates sensing offsets but collapses when actuation noise affects 15 or more of 30 workers. Hardware asymmetry across a fleet is therefore a real hazard. That study was synchronous and ran a single-robot task copied across workers. Which MAPF Model Works Best for Automated Warehousing? shows in Gazebo that stochastic physical delays break time-indexed multi-robot plans. It closes that gap with dependency graphs rather than learning. The closest prior work is ACE, which trains an asynchronous team policy with action-delay randomization, drawn uniformly from 3 to 15 steps and independently at each action. It transfers to real robots and cuts exploration time by 10.07% relative to synchronous reinforcement learning. Our proposal differs from ACE in what is randomized and how the perturbation is structured. ACE's delays are independent across actions and robots, cover only navigation, and exclude failures and robot heterogeneity; the authors list heterogeneity as open. We randomize persistent per-robot rate differences, failures, and event-order swaps. We also test whether this outcome-level randomization can stand in for physics randomization.

Core Claim

Let agent $i$ start macro-action $m$ in state $s$, and let $d(m,s)$ be the simulator's nominal duration. During training we sample the realized duration and outcome as

$$ \tilde d_i = \kappa_i \, d(m,s) \, \epsilon, \qquad \text{success} \sim \text{Bernoulli}(1-\phi_i(m)), $$

where $\kappa_i$ is a slowdown factor drawn once per robot per episode, representing persistent wear or a slow controller. The term $\epsilon$ is a mean-one jitter drawn for each execution. The quantity $\phi_i(m)$ is a per-robot failure rate for that macro-action type. On top of this, completion events from different robots that fall within a window $w$ of each other have their order randomly swapped, which models communication delay. The claim has three parts. First, team policies trained this way transfer to a physics-rich target nearly as well as policies trained with full per-robot physics randomization, at much lower simulation cost. Second, the persistent term $\kappa_i$ is the ingredient that matters: independent jitter of the same marginal size, as in ACE, will not reproduce the benefit, because real fleets are heterogeneous in a persistent way. Third, learners with timestamped agent histories such as ACAC will learn to infer a teammate's $\kappa$ from its observed durations and reassign work to it, a form of implicit online identification. Learners that discard exact timing will not.

Research Program

We will build a thin randomization layer that wraps event-driven macro-action simulators. It intercepts each macro-action's termination event and rewrites its time and outcome according to the model above. Because it acts only at termination events, it works unchanged with Mac-IAICC, ACAC and ToMacVF.

We will also build physics-rich target versions of three benchmark tasks. In these, macro-action durations are not scripted but emerge from executing the low-level controller. Box Pushing becomes two differential-drive robots pushing a rigid box with randomized friction and motor lag in a physics simulator. Warehouse Tool Delivery runs in a Gazebo warehouse in the style of Which MAPF Model Works Best for Automated Warehousing?. Overcooked gets a continuous-navigation version with physical agents. The target environments hold out physical effects never seen in training: gear backlash, a robot with degraded torque, packet loss, and stick-slip friction on the box.

The analysis has two parts. The first is a sufficiency diagnostic. We roll out scripted macro-actions in the target, fit the timing model to the observed durations and failures, and measure how much of the team policy's transfer loss disappears when the training simulator is given exactly these fitted timing statistics. If the fitted timing process explains most of the loss, the claim that timing is what matters is supported directly. The remainder measures what leaks through the abstraction, for example box pushes that fail because two robots push with mismatched force rather than at mismatched times. The second part probes the learned policies. We regress an estimate of each teammate's $\kappa$ from hidden states, and we measure behavior change, such as the fraction of tasks reassigned away from a slowed robot.

Hardware validation uses a small mobile fleet of three to four Turtlebot-class robots. One robot is deliberately degraded with a firmware torque cap and one with added command latency. The team runs Warehouse Tool Delivery and a two-robot box push.

Baselines

The naive reference point is the chosen asynchronous learner, Mac-IAICC and ACAC, trained in the nominal simulator with no randomization. This is how Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning deployed to hardware. It shows how much transfer the macro-action abstraction gives by itself. Any gain we report must be measured against it on identical target conditions.

The strongest existing method for this question is ACE-style independent action-delay randomization from ACE. It is the only published asynchronous multi-robot approach that randomizes during training and has transferred to real robots. We will port its uniform 3-to-15-step delay scheme into the same learners and simulators. We will rescale the delay range so its marginal duration variance matches our model, which isolates structure from magnitude. The file reports ACE's 10.07% real-world reduction in exploration time relative to synchronous reinforcement learning, but on exploration, not on our tasks. For Box Pushing, Overcooked and Warehouse Tool Delivery under transfer, no reported result was found on Lacuna for any asynchronous method. The published claims (ACAC, ToMacVF, Mac-IAICC) are qualitative, so we will run all baselines ourselves. Beating ACE means a statistically significant improvement in target-environment success rate and makespan over 5 seeds and 100 evaluation episodes, especially on held-out heterogeneous fleets.

The expensive upper reference is full per-robot physics randomization in the target simulator: mass, friction, motor strength and latency sampled independently for each robot, in the style of Sim-to-Real: Learning Agile Locomotion For Quadruped Robots extended to teams. Our method does not need to beat it. It needs to match it within a few points at a stated fraction of the simulation time. Both get the same environment-step budget and then the same wall-clock budget, reported separately.

Macro-level system identification is the calibration baseline. It sets each robot's timing statistics to values measured on the target and trains on those without randomization. Beating it means doing better when the test fleet differs from the calibrated one.

Three ablations isolate the ingredients. One removes the persistent factor $\kappa_i$ while keeping the variance matched. One removes failures. One removes order swaps. A further ablation replaces ACAC's timestamp encoder with padding to test the implicit-identification claim.

Why not just…?

Why not just randomize the physics of every robot and let durations emerge? This is the most serious threat, and the experiments include it as an upper reference. The argument against it is cost and dimensionality. Per-robot physics randomization multiplies the parameter space by fleet size, and it requires running the low-level controller at every step inside reinforcement learning. That removes the event-driven speed that makes macro-action learning practical: Deep Reinforcement Learning for Event-Driven Multi-Agent Decision Processes shows that fixed-step simulation needs time steps that shrink roughly as $N^{2.2}$ in the number of agents to preserve event order. If physics randomization wins by a wide margin at equal wall-clock time, our sufficiency claim is wrong, and the sufficiency diagnostic will show which effects leak.

Why not just use ACE-style delay jitter, which already transferred? Independent per-action jitter averages out over an episode. A policy trained this way can still assume all teammates are equally fast on average. The multi-robot evidence from Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning says that persistent, asymmetric actuation defects are what break shared learning. The variance-matched ablation that removes $\kappa_i$ tests exactly this. If it performs as well as the full model, we will report that persistence does not matter.

Why not just measure the real robots' task durations and put them in the simulator? Macro-level system identification is cheap and may work well for a fixed, known fleet. It fails when robots degrade, are swapped, or face conditions outside the calibration runs, which is the case the problem statement of SPiDR warns about when real dynamics leave the training support. We test it on a held-out fleet in which a different robot is degraded than the one calibrated. We also report the case where calibration wins, the matched fleet.

Why not just execute with dependency graphs, as warehouses do? Action dependency graphs, as in Which MAPF Model Works Best for Automated Warehousing?, guarantee safe ordering under delay. They do not decide who should do which task when one robot is slow. We run a hybrid in which the team policy is wrapped with a dependency-graph executor, both with and without timing randomization. This separates safety gains from allocation gains.

Proposed Experiments

The first decisive check is the cost-matched transfer comparison. For each of the three tasks, the nominal learner, ACE-style jitter, our timing randomization, macro-level calibration, and full physics randomization are all trained with Mac-IAICC and ACAC and evaluated in the physics-rich target. We report success rate, makespan, deadlock count, and transfer ratio (target return over source return), at matched environment steps and at matched wall-clock time. The prediction is that timing randomization closes most of the gap between the nominal learner and physics randomization at under a quarter of physics randomization's wall-clock cost, and that it clearly beats variance-matched jitter on heterogeneous fleets.

The second check is the sufficiency diagnostic described above, reported as the fraction of transfer loss removed by training on fitted target timing. We expect it to be high for Warehouse Tool Delivery and Overcooked and lower for contact-heavy box pushing. The box-pushing residual would indicate that force coordination leaks through the abstraction.

The third check is the implicit-identification probe. If we are right, ACAC trained with $\kappa$ randomization will decode teammate slowness from its hidden state and reassign tasks away from slow robots, while a padding-based critic will not.

The final check is the hardware run on a deliberately degraded Turtlebot fleet, with 20 trials per method for the nominal, ACE-style, and timing-randomized policies, measuring delivery makespan and failed handoffs.