When a robot trained in simulation is put on real hardware, it fails in specific places: a patch of floor that is slicker than the simulator assumed, a ramp that compresses the suspension, a motor that delivers less torque than it should. The safest way to learn about these places is to probe them carefully. The robot stays close to a known-safe fallback controller, gathers real data where the simulator is uncertain, and widens its own safety margins until it knows more. This works for one robot, but it is slow, and the samples are expensive. A fleet changes the economics. If five robots share a warehouse floor, the slick patch is the same slick patch for all of them, and one robot's careful probe could teach the other four. The catch is that some of each robot's mismatch is its own, such as a weak motor or an offset encoder. Pooling that part of the data poisons everyone else's model.
This note proposes a decentralized method for continuous multi-robot systems. The fleet splits the work of learning where the simulator is wrong into a part shared by all robots, tied to the environment, and a small part specific to each robot, tied to its hardware. The robots then decide jointly which robot should safely probe which uncertain region. Each robot's safety filter uses its own uncertainty, so a lost message can only make a robot more cautious, never less safe. The testable prediction is that the real-world samples each robot needs to reach a given safety margin fall roughly as $1/N$ in fleet size $N$ for the shared component and stay constant for the robot-specific one. Naive pooling breaks safety, and independent learning wastes the fleet.
Core Claim
The closest existing work is Provably Safe Sim-to-Real Transfer. It finds the "mismatch region" where a simulator disagrees with reality, uses real data only there, and explores with a mixture of an exploratory policy and a known-safe baseline. It reports 0% unsafe iterations and needs 77,958 real samples against 296,910 for a reward-free safe RL baseline. It is tabular and single-agent. It also assumes the simulator is either exactly right or wrong by a detectable margin, which does not hold for continuous friction or actuator drift.
Our claim differs in two ways. First, we replace the detect-and-isolate step with a continuous residual model that carries calibrated uncertainty and is split into shared and robot-specific parts. Second, we turn "where should real samples be collected" into a fleet coordination problem. Robots are assigned probe regions in the same way exploration planners assign frontiers, where a frontier is the boundary between mapped and unmapped space.
The second ingredient also addresses a tension the file documents. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that safer policies stop visiting the safety boundary, so the estimate of where the boundary lies degrades. Blind Spot Detection for Safe Sim-to-Real Transfer finds the mirror image: mistake-prone data mapped blind spots with AUC 0.96, versus 0.60 from an optimal oracle. Deliberate, shielded boundary probing, spread across a fleet, is a way to collect that boundary data without any robot making real mistakes.
Research Program
Residual dynamics model. We model real dynamics as the simulator plus two learned corrections:
$$f_{\text{real}}^{i}(s,a) = f_{\text{sim}}(s,a) + g(x, s, a) + h_{\theta_i}(s,a).$$
The symbols are as follows:
- $f_{\text{real}}^{i}$ is robot $i$'s true next-state map.
- $f_{\text{sim}}$ is the simulator.
- $g$ is a shared residual that depends on workspace location $x$ and the local state-action. It captures terrain and contact effects.
- $h_{\theta_i}$ is a low-dimensional robot-specific residual with parameters $\theta_i$, such as actuator gain and offset per joint or wheel.
The deliberately low dimension of $\theta_i$ is what makes the split identifiable. Hardware faults must be explained by a few numbers, and anything spatially structured is pushed into $g$. Each robot keeps an uncertainty-aware model of both parts, either an ensemble or a sparse Gaussian process (a model that returns a standard deviation alongside each prediction).
Safety filter. Each robot runs a control barrier function filter, which minimally edits the commanded action so a safety function stays nonnegative. The filter covers obstacles and neighboring robots. Its margin is inflated by $\beta\,\sigma_i(s,a)$, where $\sigma_i$ is the robot's current predictive uncertainty and $\beta$ is a confidence multiplier. This is the continuous analogue of the tabular mismatch-region guarantee.
The design takes its structure from Robust and Safe Multi-Agent Reinforcement Learning Framework with Communication for Autonomous Vehicles, which places a barrier-function filter between a multi-agent policy and the actuators. The difference is that our margin shrinks as data arrives, while theirs is fixed and hand-specified. It also differs from the online barrier functions of Provably Safe Online Multi-Agent Navigation in Unknown Environments. That work learns the geometry of free space from LiDAR scans but treats robot dynamics as known.
Safe probing. A probe is a short excursion toward an uncertain state-action region. The excursion is accepted only if the uncertainty-inflated filter certifies that the fallback controller can still recover from its endpoint. This is the mixture-policy idea from the tabular paper, with the mixing weight replaced by a reachability check on the inflated model.
Coordinated allocation. The fleet repeatedly decides which robot probes which region. The objective is the expected reduction in shared uncertainty $g$, weighted by how much that region constrains the task (how often the task policy wants to go there), minus travel cost and minus task reward forgone. Allocation runs as decentralized auction-style task assignment, following the style of Behaviorally Heterogeneous Multi-Agent Exploration Using Distributed Task Allocation. Robot-specific uncertainty in $h_{\theta_i}$ can only be reduced by robot $i$ itself, so it enters as a private term in that robot's bid.
Communication. Robots exchange probe data, meaning transitions and summary statistics, asynchronously. They do not exchange intended actions. This is a deliberate contrast with DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning, whose constraint satisfaction depends on synchronous broadcast of neighbors' base actions at every step.
In our design, a dropped packet means a robot's posterior over $g$ is less informed and its margins stay wider. Safety therefore degrades into conservatism rather than into violation, provided the uncertainty estimates are valid. Inter-robot avoidance uses onboard sensing of neighbors' positions.
Quarantine test. Before a robot's transitions are fused into $g$, the robot checks whether its residuals are explained by its own $h_{\theta_i}$ plus the current $g$. If they are not, the robot is quarantined and its data stays local. This test targets a failure documented in Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning. There, shared learning across 30 arms absorbed sensing offsets but collapsed under variable actuation noise, dropping to a mean score of about −6 versus about −2. Collapse occurred once roughly half the agents were affected. Our design takes that negative result as a constraint: shared models must not absorb per-robot actuation faults.
Theory. We will prove a sample-sharing bound. Under a Gaussian-process prior on $g$ with bounded information gain, the sample requirement has the form stated below. The form follows from standard information-gain bounds applied to the pooled shared data and to each robot's private parameters separately. We will also prove the monotonicity property: withholding messages never shrinks any robot's margin below its local-data margin.
The per-robot real-sample requirement for margin error $\le \eta$ is bounded by
$$\frac{C_g(\eta)}{N} + C_h(\eta, \dim\theta).$$
Here:
- $C_g(\eta)$ is the single-robot cost of learning the shared residual.
- $C_h(\eta, \dim\theta)$ is the cost of the robot-specific part, which grows with the parameter count rather than with workspace size.
Baselines
Domain randomization with a fixed margin. The naive reference point is a multi-agent policy trained with domain randomization over friction, mass, latency, and actuator gain. Domain randomization means training across many randomly perturbed simulator settings. The policy is deployed with a fixed barrier-function filter whose margin is set by hand, at the most conservative value that yields zero violations on the test variants.
This mirrors the robust-training-plus-fixed-filter design of Robust and Safe Multi-Agent Reinforcement Learning Framework with Communication for Autonomous Vehicles. That paper reports a 0% collision rate on 1/10th-scale hardware and a filter intervention rate falling from 18.7% to 11.4% when vehicle-to-vehicle communication is enabled. It is the strongest existing safe multi-robot transfer system in the file. Because it is already at zero collisions, beating it cannot mean fewer collisions. It must mean matching zero violations while recovering task performance lost to the fixed margin, and needing no hand tuning.
We give it the same simulator budget and the same task policy, and tune its margin on the same validation variants. We report its intervention rate and task return beside ours.
Independent per-robot adaptation. This is the single-agent mismatch-learning recipe run on each robot separately: same residual model, same filter, same probing rule, no sharing. It is the continuous, per-robot reading of Provably Safe Sim-to-Real Transfer. Its reported tabular figures are 0% unsafe iterations and 77,958 real samples; we cannot reproduce that exact setting in continuous control. This baseline isolates the value of sharing. Beating it means reaching the same margin error with fewer real samples per robot, at no cost in violations, with the gap widening as $N$ grows.
Naive pooling. One residual model is fit on all robots' data with no robot-specific term and no quarantine. This is the ablation of the shared/specific split. Beating it means avoiding the violations or over-conservatism it should suffer when a minority of robots carry actuation faults.
Passive sharing. Sharing and the split are kept, but coordinated probing is removed. Robots learn only from transitions their task policy happens to produce. This isolates the allocation ingredient. Beating it means reaching the target margin with fewer samples, or fewer task-time interruptions.
Constrained multi-agent RL. As a learning-side reference, we run MAPPO-Lagrangian from Multi-Agent Constrained Policy Optimisation, using the released code (chauncygu codebase). It is trained with domain randomization and has no filter. That paper reports near-zero cost on Safe Multi-Agent MuJoCo but no tabulated numbers; we found no reported transfer result on Lacuna for it.
All learned methods receive equal simulator steps, equal real-sample budgets, and an equal hyperparameter search of 20 configurations each.
Why not just…?
Why not just randomize more widely and keep a conservative margin? This is the strongest threat, because it already achieves zero hardware collisions in Robust and Safe Multi-Agent Reinforcement Learning Framework with Communication for Autonomous Vehicles. Its cost is conservatism: a margin wide enough for the worst randomized case is too wide almost everywhere else, and the file lists over-conservatism as a known limitation of worst-case formulations.
The experiment settles this directly. We measure task return and filter intervention rate at matched zero-violation operating points. If the fixed margin recovers within a few percent of our task return, adaptation is not worth its complexity in that regime. We would then report that result and restrict our claim to settings with strongly localized mismatch.
Why not just let each robot adapt on its own? It is safe, simple, and needs no communication. It ignores the fact that most mismatch, such as floors, ramps, and contact surfaces, is shared. The prediction is that its per-robot sample cost stays flat as the fleet grows, while ours falls toward the robot-specific floor $C_h$. If our measured curve does not bend with $N$, the sharing claim fails.
Why not just pool all robots' data into one correction? It captures sharing for free. The collaborative Kuka study Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning shows that shared learning breaks under variable actuation faults on a sizable fraction of agents. We sweep the fraction of faulty robots from 0 to 50% using that paper's four perturbation types. The test is whether pooled models produce violations or inflated margins on healthy robots that the split model avoids. If pooling is harmless at realistic fault rates, the robot-specific term is unnecessary.
Why not just learn from whatever data the task produces? If the task already drives robots through the uncertain regions, coordinated probing is redundant. The safety paradox and the blind-spot findings suggest otherwise: safe task policies avoid exactly the boundary states that matter. This is still an empirical question per environment. We will report environments where passive sharing suffices, alongside those where it does not.
Proposed Experiments
Testbed. We build continuous fleet tasks on Safety-Gymnasium's multi-agent environments and Safe Multi-Agent MuJoCo (Safety Gymnasium; Multi-Agent Constrained Policy Optimisation). Two to eight point or car robots cover an arena under a time budget. A held-out "real" simulator differs from the training simulator in two ways:
- spatially localized low-friction and high-drag patches, which are shared mismatch;
- per-robot fixed and variable sensing and actuation perturbations following the four NoisyKukaReacher types, which are robot-specific mismatch.
Building a perturbed simulator as a stand-in for reality follows the evaluation logic of the tabular sim-to-real paper, scaled to continuous control.
Metrics. We report:
- real samples per robot to reach margin error $\eta$ relative to an oracle that knows the true dynamics;
- constraint violations during probing and deployment, against obstacles and between robots;
- normalized task reward and cost;
- filter intervention rate;
- coverage within the time budget.
Decisive checks.
- Fleet-size scaling. We plot per-robot samples against $N$ in {1, 2, 4, 8}, separately for shared-only and mixed mismatch. The prediction is a roughly $1/N$ decline that flattens at the robot-specific floor. A flat curve falsifies the core claim.
- Fault-fraction sweep. The sweep from 0 to 50% faulty robots compares the split model against naive pooling. Our method should show no rise in violations on healthy robots.
- Communication degradation. We apply packet loss from 0 to 90% and random blackouts. We expect zero added violations and a graceful rise in conservatism. Any violation that appears only under message loss would refute the monotonicity claim.
- Matched zero-violation comparison. We compare task return against the domain-randomized fixed-margin baseline at matched zero violations.
- Hardware (if access permits). We repeat checks 1 and 3 on four 1/10th-scale cars on a floor with taped low-friction regions. The protocol mirrors the hardware setting of the autonomous-vehicle paper, so the reported 0% collision and 11.4% intervention figures are a direct reference.
If checks 1 and 4 both come out in our favor, the fleet becomes a way to discover the sim-to-real gap safely, not just a set of robots that each must be transferred separately.
References
- Provably Safe Sim-to-Real Transfer
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Blind Spot Detection for Safe Sim-to-Real Transfer
- Robust and Safe Multi-Agent Reinforcement Learning Framework with Communication for Autonomous Vehicles
- Provably Safe Online Multi-Agent Navigation in Unknown Environments
- Behaviorally Heterogeneous Multi-Agent Exploration Using Distributed Task Allocation
- DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learning
- Towards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning
- Multi-Agent Constrained Policy Optimisation
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark